[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:09:32 dist-test-slave-14ww krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:09:32 dist-test-slave-14ww krb5kdc[399](info): set up 2 sockets
Sep 24 07:09:32 dist-test-slave-14ww krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 07:09:33.452792 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:09:33 dist-test-slave-14ww krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:09:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:09:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:09:33.568377 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42915
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42915
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:33.648653 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:33.648955 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:33.648996 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:33.650200 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:09:33.650269 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:33.650291 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:33.650312 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:09:33.650341 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:09:33.651866 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:33.652137 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:33.652390 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:33.657565 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:33.657606 412 server_base.cc:1034] running on GCE node
W20260924 07:09:33.657574 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:33.657640 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:33.658030 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:33.658315 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:33.659449 412 hybrid_clock.cc:648] HybridClock initialized: now 1790233773659428 us; error 39 us; skew 500 ppm
Sep 24 07:09:33 dist-test-slave-14ww krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233773, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:33.661825 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:09:33.662273 412 webserver.cc:460] Webserver started at http://127.0.98.62:46693/ using document root <none> and password file <none>
I20260924 07:09:33.662472 412 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:33.662513 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:33.662599 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:33.663610 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "e8a30e5fc1274b50bfd37c89fdc9d55d"
format_stamp: "Formatted at 2026-09-24 07:09:33 on dist-test-slave-14ww"
server_key: "895a30241cac8b45ee1b6e6e81e1346f"
server_key_iv: "0a6382b13258e7b0be51e4deba215100"
server_key_version: "encryptionkey@0"
I20260924 07:09:33.663919 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "e8a30e5fc1274b50bfd37c89fdc9d55d"
format_stamp: "Formatted at 2026-09-24 07:09:33 on dist-test-slave-14ww"
server_key: "895a30241cac8b45ee1b6e6e81e1346f"
server_key_iv: "0a6382b13258e7b0be51e4deba215100"
server_key_version: "encryptionkey@0"
I20260924 07:09:33.665335 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:09:33.666172 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:33.666375 412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:09:33.666460 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "e8a30e5fc1274b50bfd37c89fdc9d55d"
format_stamp: "Formatted at 2026-09-24 07:09:33 on dist-test-slave-14ww"
server_key: "895a30241cac8b45ee1b6e6e81e1346f"
server_key_iv: "0a6382b13258e7b0be51e4deba215100"
server_key_version: "encryptionkey@0"
I20260924 07:09:33.666535 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:33.684162 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:33.686187 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:33.686357 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:33.690176 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42915
I20260924 07:09:33.690208 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42915 every 8 connection(s)
I20260924 07:09:33.690570 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:09:33.691430 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:33.692520 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 412
I20260924 07:09:33.692613 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:09:33.693262 392 external_mini_cluster.cc:1428] Setting key a3701a0e3686a16fc4314444abcb1e45
I20260924 07:09:33.693970 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8a30e5fc1274b50bfd37c89fdc9d55d: Bootstrap starting.
I20260924 07:09:33.694644 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8a30e5fc1274b50bfd37c89fdc9d55d: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:33.694921 480 log.cc:824] T 00000000000000000000000000000000 P e8a30e5fc1274b50bfd37c89fdc9d55d: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:33.695568 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8a30e5fc1274b50bfd37c89fdc9d55d: No bootstrap required, opened a new log
I20260924 07:09:33.697189 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8a30e5fc1274b50bfd37c89fdc9d55d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8a30e5fc1274b50bfd37c89fdc9d55d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42915 } }
I20260924 07:09:33.697348 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8a30e5fc1274b50bfd37c89fdc9d55d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:33.697410 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8a30e5fc1274b50bfd37c89fdc9d55d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8a30e5fc1274b50bfd37c89fdc9d55d, State: Initialized, Role: FOLLOWER
I20260924 07:09:33.697539 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8a30e5fc1274b50bfd37c89fdc9d55d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8a30e5fc1274b50bfd37c89fdc9d55d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42915 } }
I20260924 07:09:33.697630 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8a30e5fc1274b50bfd37c89fdc9d55d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:09:33.697674 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8a30e5fc1274b50bfd37c89fdc9d55d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:09:33.697721 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8a30e5fc1274b50bfd37c89fdc9d55d [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:33.698523 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8a30e5fc1274b50bfd37c89fdc9d55d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8a30e5fc1274b50bfd37c89fdc9d55d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42915 } }
I20260924 07:09:33.698634 480 leader_election.cc:302] T 00000000000000000000000000000000 P e8a30e5fc1274b50bfd37c89fdc9d55d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8a30e5fc1274b50bfd37c89fdc9d55d; no voters:
I20260924 07:09:33.698804 480 leader_election.cc:288] T 00000000000000000000000000000000 P e8a30e5fc1274b50bfd37c89fdc9d55d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:09:33.698825 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8a30e5fc1274b50bfd37c89fdc9d55d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:09:33.698933 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8a30e5fc1274b50bfd37c89fdc9d55d [term 1 LEADER]: Becoming Leader. State: Replica: e8a30e5fc1274b50bfd37c89fdc9d55d, State: Running, Role: LEADER
I20260924 07:09:33.699014 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8a30e5fc1274b50bfd37c89fdc9d55d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8a30e5fc1274b50bfd37c89fdc9d55d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42915 } }
I20260924 07:09:33.699066 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8a30e5fc1274b50bfd37c89fdc9d55d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:09:33.699383 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8a30e5fc1274b50bfd37c89fdc9d55d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8a30e5fc1274b50bfd37c89fdc9d55d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8a30e5fc1274b50bfd37c89fdc9d55d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42915 } } }
I20260924 07:09:33.699530 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8a30e5fc1274b50bfd37c89fdc9d55d [sys.catalog]: This master's current role is: LEADER
I20260924 07:09:33.699786 490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:09:33.700279 490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:09:33.700687 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8a30e5fc1274b50bfd37c89fdc9d55d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8a30e5fc1274b50bfd37c89fdc9d55d. Latest consensus state: current_term: 1 leader_uuid: "e8a30e5fc1274b50bfd37c89fdc9d55d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8a30e5fc1274b50bfd37c89fdc9d55d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42915 } } }
I20260924 07:09:33.700852 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8a30e5fc1274b50bfd37c89fdc9d55d [sys.catalog]: This master's current role is: LEADER
I20260924 07:09:33.701963 490 catalog_manager.cc:1347] Generated new cluster ID: 52cf868956104cb59252178f2d65b0d6
I20260924 07:09:33.702018 490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 07:09:33 dist-test-slave-14ww krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:09:33.710868 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:33.694172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51948 (local address 127.0.98.62:42915)
0924 07:09:33.694332 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:33.694342 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:33.694613 (+ 271us) server_negotiation.cc:407] Connection header received
0924 07:09:33.697200 (+ 2587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:33.697207 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:33.697254 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:33.697424 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:33.699100 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:33.700588 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:33.702085 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:33.702666 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:33.702786 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:33.706841 (+ 4055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:33.706845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:33.706846 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:33.706863 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:33.708845 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:33.709338 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:33.709340 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:33.709341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:33.709456 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:33.709733 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:33.709733 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:33.709734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:33.710019 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:33.710181 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:33.710527 (+ 346us) server_negotiation.cc:299] Negotiation successful
0924 07:09:33.710654 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260924 07:09:33.712709 490 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:09:33.713418 490 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:09:33.716257 490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8a30e5fc1274b50bfd37c89fdc9d55d: Generated new TSK 0
I20260924 07:09:33.716408 490 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 07:09:33.760999 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42915
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:33.840898 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:33.841089 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:33.841110 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:33.842154 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:33.842188 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:33.842214 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:09:33.843540 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42915
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:33.843705 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:33.843868 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:33.848654 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:33.848711 506 server_base.cc:1034] running on GCE node
W20260924 07:09:33.848616 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:33.848623 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:33.849004 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:33.849192 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:33.850366 506 hybrid_clock.cc:648] HybridClock initialized: now 1790233773850348 us; error 35 us; skew 500 ppm
Sep 24 07:09:33 dist-test-slave-14ww krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233773, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:33.852691 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:09:33.853111 506 webserver.cc:460] Webserver started at http://127.0.98.1:37267/ using document root <none> and password file <none>
I20260924 07:09:33.853283 506 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:33.853322 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:33.853410 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:33.854369 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-0/data/instance:
uuid: "da6d835fece148b88d5c0358c09c38fd"
format_stamp: "Formatted at 2026-09-24 07:09:33 on dist-test-slave-14ww"
server_key: "c58a87d79d103cd1a7e048e6dd553c89"
server_key_iv: "73abeded965dcae681a4c8de2c4980dc"
server_key_version: "encryptionkey@0"
I20260924 07:09:33.854669 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-0/wal/instance:
uuid: "da6d835fece148b88d5c0358c09c38fd"
format_stamp: "Formatted at 2026-09-24 07:09:33 on dist-test-slave-14ww"
server_key: "c58a87d79d103cd1a7e048e6dd553c89"
server_key_iv: "73abeded965dcae681a4c8de2c4980dc"
server_key_version: "encryptionkey@0"
I20260924 07:09:33.855856 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 07:09:33.856498 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:33.856653 506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:33.856721 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-0/wal
uuid: "da6d835fece148b88d5c0358c09c38fd"
format_stamp: "Formatted at 2026-09-24 07:09:33 on dist-test-slave-14ww"
server_key: "c58a87d79d103cd1a7e048e6dd553c89"
server_key_iv: "73abeded965dcae681a4c8de2c4980dc"
server_key_version: "encryptionkey@0"
I20260924 07:09:33.856833 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:33.879943 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:33.882189 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:33.882344 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:33.882548 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:33.882833 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:33.882882 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:33.882921 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:33.882948 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:33.888679 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39559
I20260924 07:09:33.888789 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39559 every 8 connection(s)
I20260924 07:09:33.889058 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 07:09:33 dist-test-slave-14ww krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233773, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:09:33.895707 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 506
I20260924 07:09:33.895817 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:09:33.895972 392 external_mini_cluster.cc:1428] Setting key efa0adfdb73a16fb8dca62ccf77f16a3
I20260924 07:09:33.898000 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:33.889733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39831 (local address 127.0.98.62:42915)
0924 07:09:33.889778 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:33.889780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:33.889989 (+ 209us) server_negotiation.cc:407] Connection header received
0924 07:09:33.890460 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:33.890461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:33.890475 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:33.890527 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:33.891379 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:33.891880 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:33.892891 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:33.893084 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:33.893139 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:33.895047 (+ 1908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:33.895049 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:33.895049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:33.895055 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:33.896621 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:33.897061 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:33.897062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:33.897063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:33.897122 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:33.897373 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:33.897374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:33.897374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:33.897485 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:33.897553 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:33.897823 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 07:09:33.897911 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 07:09:33.898180 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:33.889698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42915 (local address 127.0.98.1:39831)
0924 07:09:33.889844 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:33.889877 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:33.890375 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:33.890577 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:33.890580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:33.891119 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:33.891304 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:33.891309 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:33.891951 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:33.891951 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:33.892809 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:33.892811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:33.893149 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:33.893150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:33.893248 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:33.893659 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:33.893666 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:33.894960 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:33.896725 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:33.896773 (+ 48us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:33.896775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:33.896985 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:33.897186 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:33.897187 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:33.897187 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:33.897307 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:33.897555 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:33.897561 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:33.897691 (+ 130us) client_negotiation.cc:769] Sending connection context
0924 07:09:33.897788 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 07:09:33.897936 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":88,"threads_started":1}
I20260924 07:09:33.898882 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42915
I20260924 07:09:33.898989 635 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:33.899150 635 heartbeater.cc:511] Master 127.0.98.62:42915 requested a full tablet report, sending...
I20260924 07:09:33.899679 444 ts_manager.cc:194] Registered new tserver with Master: da6d835fece148b88d5c0358c09c38fd (127.0.98.1:39559)
I20260924 07:09:33.900396 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39831
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 07:09:33.920688 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42915
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:34.004525 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:34.004733 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:34.004792 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:34.005910 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:34.005973 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:34.006021 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:09:34.007467 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42915
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:34.007712 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:34.007926 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:34.012902 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:34.012908 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:34.012998 640 server_base.cc:1034] running on GCE node
W20260924 07:09:34.012908 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:34.013396 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:34.013625 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:34.014796 640 hybrid_clock.cc:648] HybridClock initialized: now 1790233774014778 us; error 38 us; skew 500 ppm
Sep 24 07:09:34 dist-test-slave-14ww krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:34.017369 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:09:34.017726 640 webserver.cc:460] Webserver started at http://127.0.98.2:38569/ using document root <none> and password file <none>
I20260924 07:09:34.017959 640 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:34.018004 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:34.018152 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:34.019150 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-1/data/instance:
uuid: "e3d809214264403e89348e333014e226"
format_stamp: "Formatted at 2026-09-24 07:09:34 on dist-test-slave-14ww"
server_key: "8ff02b1d0286ec2e603d795bc522ddc1"
server_key_iv: "873e9436924fca123e2b593c90ff9284"
server_key_version: "encryptionkey@0"
I20260924 07:09:34.019491 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e3d809214264403e89348e333014e226"
format_stamp: "Formatted at 2026-09-24 07:09:34 on dist-test-slave-14ww"
server_key: "8ff02b1d0286ec2e603d795bc522ddc1"
server_key_iv: "873e9436924fca123e2b593c90ff9284"
server_key_version: "encryptionkey@0"
I20260924 07:09:34.020957 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 07:09:34.021677 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:34.021874 640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:09:34.021938 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-1/wal
uuid: "e3d809214264403e89348e333014e226"
format_stamp: "Formatted at 2026-09-24 07:09:34 on dist-test-slave-14ww"
server_key: "8ff02b1d0286ec2e603d795bc522ddc1"
server_key_iv: "873e9436924fca123e2b593c90ff9284"
server_key_version: "encryptionkey@0"
I20260924 07:09:34.022018 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:34.028128 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:34.029963 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:34.030095 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:34.030301 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:34.030567 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:34.030599 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:34.030635 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:34.030664 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:34.036968 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33381
I20260924 07:09:34.037034 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33381 every 8 connection(s)
I20260924 07:09:34.037331 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 07:09:34 dist-test-slave-14ww krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:09:34.044803 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 640
I20260924 07:09:34.044926 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:09:34.045114 392 external_mini_cluster.cc:1428] Setting key a5da013728acc6044a175371ef08f7eb
I20260924 07:09:34.046097 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:34.038080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40705 (local address 127.0.98.62:42915)
0924 07:09:34.038143 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:34.038145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:34.038380 (+ 235us) server_negotiation.cc:407] Connection header received
0924 07:09:34.038819 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:34.038820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:34.038830 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:34.038871 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:34.039714 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:34.040213 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:34.041267 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:34.041457 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:34.041524 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:34.043408 (+ 1884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:34.043410 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:34.043410 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:34.043417 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:34.044631 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:34.045114 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:34.045115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:34.045116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:34.045177 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:34.045466 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:34.045467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:34.045467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:34.045593 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:34.045660 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:34.045963 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 07:09:34.046034 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 07:09:34.046427 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:34.038081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42915 (local address 127.0.98.2:40705)
0924 07:09:34.038257 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:34.038298 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:34.038732 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:34.038921 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:34.038924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:34.039464 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:34.039642 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:34.039646 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:34.040287 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:34.040288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:34.041176 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:34.041178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:34.041533 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:34.041534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:34.041628 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:34.042050 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:34.042058 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:34.043328 (+ 1270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:34.044705 (+ 1377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:34.044712 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:34.044714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:34.045037 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:34.045244 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:34.045245 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:34.045245 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:34.045392 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:34.045655 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:34.045672 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:34.045829 (+ 157us) client_negotiation.cc:769] Sending connection context
0924 07:09:34.045950 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 07:09:34.046122 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":99,"threads_started":1}
I20260924 07:09:34.046886 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42915
I20260924 07:09:34.046988 769 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:34.047191 769 heartbeater.cc:511] Master 127.0.98.62:42915 requested a full tablet report, sending...
I20260924 07:09:34.047574 444 ts_manager.cc:194] Registered new tserver with Master: e3d809214264403e89348e333014e226 (127.0.98.2:33381)
I20260924 07:09:34.047940 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40705
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 07:09:34.076176 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42915
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:34.155618 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:34.155845 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:34.155884 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:34.157058 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:34.157119 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:34.157169 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:09:34.158573 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42915
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:34.158820 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:34.159034 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:34.163971 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:34.164029 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:34.164165 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:34.164325 774 server_base.cc:1034] running on GCE node
I20260924 07:09:34.164511 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:34.164762 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:34.165913 774 hybrid_clock.cc:648] HybridClock initialized: now 1790233774165901 us; error 29 us; skew 500 ppm
Sep 24 07:09:34 dist-test-slave-14ww krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:34.168303 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:09:34.168799 774 webserver.cc:460] Webserver started at http://127.0.98.3:34321/ using document root <none> and password file <none>
I20260924 07:09:34.169068 774 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:34.169114 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:34.169247 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:34.170320 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-2/data/instance:
uuid: "5ce2d50fa7694b1bb238aa88de9db255"
format_stamp: "Formatted at 2026-09-24 07:09:34 on dist-test-slave-14ww"
server_key: "b974e997b4c21cce1dc98e5b5673b7d7"
server_key_iv: "266f56d8529a72b5c829221cb14b2da1"
server_key_version: "encryptionkey@0"
I20260924 07:09:34.170667 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5ce2d50fa7694b1bb238aa88de9db255"
format_stamp: "Formatted at 2026-09-24 07:09:34 on dist-test-slave-14ww"
server_key: "b974e997b4c21cce1dc98e5b5673b7d7"
server_key_iv: "266f56d8529a72b5c829221cb14b2da1"
server_key_version: "encryptionkey@0"
I20260924 07:09:34.171911 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:09:34.172585 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:34.172796 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:34.172888 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-2/wal
uuid: "5ce2d50fa7694b1bb238aa88de9db255"
format_stamp: "Formatted at 2026-09-24 07:09:34 on dist-test-slave-14ww"
server_key: "b974e997b4c21cce1dc98e5b5673b7d7"
server_key_iv: "266f56d8529a72b5c829221cb14b2da1"
server_key_version: "encryptionkey@0"
I20260924 07:09:34.172960 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:34.182226 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:34.183923 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:34.184072 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:34.184280 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:34.184563 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:34.184614 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:34.184653 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:34.184681 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:34.191087 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42377
I20260924 07:09:34.191148 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42377 every 8 connection(s)
I20260924 07:09:34.191448 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 07:09:34 dist-test-slave-14ww krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:09:34.200416 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 774
I20260924 07:09:34.200517 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:34.192136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45655 (local address 127.0.98.62:42915)
0924 07:09:34.192193 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:34.192195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:34.192386 (+ 191us) server_negotiation.cc:407] Connection header received
0924 07:09:34.193410 (+ 1024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:34.193417 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:34.193430 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:34.193487 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:34.194078 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:34.194620 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:34.195620 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:34.195812 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:34.195894 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:34.197877 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:34.197878 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:34.197879 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:34.197887 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:34.199067 (+ 1180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:34.199535 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:34.199536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:34.199536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:34.199596 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:34.199839 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:34.199840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:34.199840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:34.199949 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:34.200023 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:34.200322 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 07:09:34.200427 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 07:09:34.200532 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790233772407728-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:09:34.200651 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:34.192114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42915 (local address 127.0.98.3:45655)
0924 07:09:34.192267 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:34.192303 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:34.193300 (+ 997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:34.193563 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:34.193567 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:34.193807 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:34.193985 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:34.193989 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:34.194728 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:34.194729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:34.195533 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:34.195536 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:34.195902 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:34.195903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:34.196019 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:34.196456 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:34.196464 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:34.197786 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:34.199179 (+ 1393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:34.199185 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:34.199187 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:34.199464 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:34.199666 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:34.199667 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:34.199667 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:34.199778 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:34.200012 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:34.200039 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:34.200163 (+ 124us) client_negotiation.cc:769] Sending connection context
0924 07:09:34.200290 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 07:09:34.200440 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
I20260924 07:09:34.200712 392 external_mini_cluster.cc:1428] Setting key 935ec3bd9ee836e437e3a4717c599dfd
I20260924 07:09:34.201066 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42915
I20260924 07:09:34.201154 903 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:34.201345 903 heartbeater.cc:511] Master 127.0.98.62:42915 requested a full tablet report, sending...
I20260924 07:09:34.201699 444 ts_manager.cc:194] Registered new tserver with Master: 5ce2d50fa7694b1bb238aa88de9db255 (127.0.98.3:42377)
I20260924 07:09:34.202108 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45655
I20260924 07:09:34.212790 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 07:09:34 dist-test-slave-14ww krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233774, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 07:09:34 dist-test-slave-14ww krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233774, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 07:09:34.235256 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:34.227805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40096 (local address 127.0.98.1:39559)
0924 07:09:34.227948 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:34.227958 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:34.227968 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:09:34.227992 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:34.227993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:34.228006 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:34.228081 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:34.228575 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:34.229389 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:34.230118 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:34.230470 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:34.230530 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:34.232639 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:34.232645 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:34.232646 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:34.232652 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:34.234218 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:34.234532 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:34.234535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:34.234535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:34.234600 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:34.234764 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:34.234764 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:34.234764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:34.234893 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:34.234993 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:34.235068 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 07:09:34.235127 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":83,"threads_started":1}
W20260924 07:09:34.235617 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40096
Sep 24 07:09:34 dist-test-slave-14ww krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 07:09:34 dist-test-slave-14ww krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 07:09:34.256145 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:34.249491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40112 (local address 127.0.98.1:39559)
0924 07:09:34.249563 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:34.249565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:34.249639 (+ 74us) server_negotiation.cc:407] Connection header received
0924 07:09:34.249715 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:34.249716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:34.249733 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:34.249798 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:34.250351 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:34.250821 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:34.251529 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:34.251724 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:34.251806 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:34.253914 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:34.253916 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:34.253916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:34.253925 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:34.255098 (+ 1173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:34.255432 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:34.255433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:34.255434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:34.255487 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:34.255667 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:34.255668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:34.255668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:34.255760 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:34.255884 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:34.255963 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 07:09:34.256047 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 07:09:34.256646 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 506
I20260924 07:09:34.261092 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 640
I20260924 07:09:34.266745 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 774
I20260924 07:09:34.271149 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 412
2026-09-24T07:09:34Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1870 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:09:34 dist-test-slave-14ww krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:09:34 dist-test-slave-14ww krb5kdc[917](info): set up 2 sockets
Sep 24 07:09:34 dist-test-slave-14ww krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260924 07:09:35.316234 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:09:35 dist-test-slave-14ww krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:09:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:09:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:09:35.432118 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46321
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46321
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:35.511446 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:35.511674 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:35.511710 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:35.512857 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:09:35.512924 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:35.512959 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:35.512987 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:09:35.513010 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:09:35.514461 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:35.514726 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:35.514945 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:35.519650 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:35.519740 930 server_base.cc:1034] running on GCE node
W20260924 07:09:35.519696 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:35.519649 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:35.520113 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:35.520426 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:35.521600 930 hybrid_clock.cc:648] HybridClock initialized: now 1790233775521588 us; error 27 us; skew 500 ppm
Sep 24 07:09:35 dist-test-slave-14ww krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:35.523952 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:09:35.524410 930 webserver.cc:460] Webserver started at http://127.0.98.62:35251/ using document root <none> and password file <none>
I20260924 07:09:35.524632 930 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:35.524695 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:35.524871 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:35.525848 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "695550a44a414e92b6ab450443b3910c"
format_stamp: "Formatted at 2026-09-24 07:09:35 on dist-test-slave-14ww"
server_key: "b9a180c0b9454afa5fcdacb051c7ddd8"
server_key_iv: "47549aa2684d022f5c42c16aa636c468"
server_key_version: "encryptionkey@0"
I20260924 07:09:35.526232 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "695550a44a414e92b6ab450443b3910c"
format_stamp: "Formatted at 2026-09-24 07:09:35 on dist-test-slave-14ww"
server_key: "b9a180c0b9454afa5fcdacb051c7ddd8"
server_key_iv: "47549aa2684d022f5c42c16aa636c468"
server_key_version: "encryptionkey@0"
I20260924 07:09:35.527487 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 07:09:35.528246 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:35.528437 930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:09:35.528530 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "695550a44a414e92b6ab450443b3910c"
format_stamp: "Formatted at 2026-09-24 07:09:35 on dist-test-slave-14ww"
server_key: "b9a180c0b9454afa5fcdacb051c7ddd8"
server_key_iv: "47549aa2684d022f5c42c16aa636c468"
server_key_version: "encryptionkey@0"
I20260924 07:09:35.528604 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:35.539356 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:35.541220 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:35.541365 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:35.545140 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46321
I20260924 07:09:35.545186 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46321 every 8 connection(s)
I20260924 07:09:35.545521 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:09:35.546049 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 930
I20260924 07:09:35.546142 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:09:35.546308 392 external_mini_cluster.cc:1428] Setting key 938baaea936f60d075e7869a7bedf7f2
I20260924 07:09:35.546486 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:35.549136 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 695550a44a414e92b6ab450443b3910c: Bootstrap starting.
I20260924 07:09:35.549682 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 695550a44a414e92b6ab450443b3910c: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:35.549945 998 log.cc:824] T 00000000000000000000000000000000 P 695550a44a414e92b6ab450443b3910c: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:35.550566 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 695550a44a414e92b6ab450443b3910c: No bootstrap required, opened a new log
Sep 24 07:09:35 dist-test-slave-14ww krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:09:35.552719 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 695550a44a414e92b6ab450443b3910c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695550a44a414e92b6ab450443b3910c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46321 } }
I20260924 07:09:35.552917 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 695550a44a414e92b6ab450443b3910c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:35.552965 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 695550a44a414e92b6ab450443b3910c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 695550a44a414e92b6ab450443b3910c, State: Initialized, Role: FOLLOWER
I20260924 07:09:35.553099 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 695550a44a414e92b6ab450443b3910c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695550a44a414e92b6ab450443b3910c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46321 } }
I20260924 07:09:35.553190 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 695550a44a414e92b6ab450443b3910c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:09:35.553260 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 695550a44a414e92b6ab450443b3910c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:09:35.553308 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 695550a44a414e92b6ab450443b3910c [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:35.553948 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 695550a44a414e92b6ab450443b3910c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695550a44a414e92b6ab450443b3910c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46321 } }
I20260924 07:09:35.554056 998 leader_election.cc:302] T 00000000000000000000000000000000 P 695550a44a414e92b6ab450443b3910c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 695550a44a414e92b6ab450443b3910c; no voters:
I20260924 07:09:35.554205 998 leader_election.cc:288] T 00000000000000000000000000000000 P 695550a44a414e92b6ab450443b3910c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:09:35.554292 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 695550a44a414e92b6ab450443b3910c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:09:35.554459 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 695550a44a414e92b6ab450443b3910c [term 1 LEADER]: Becoming Leader. State: Replica: 695550a44a414e92b6ab450443b3910c, State: Running, Role: LEADER
I20260924 07:09:35.554484 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 695550a44a414e92b6ab450443b3910c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:09:35.554606 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 695550a44a414e92b6ab450443b3910c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695550a44a414e92b6ab450443b3910c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46321 } }
I20260924 07:09:35.555887 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 695550a44a414e92b6ab450443b3910c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "695550a44a414e92b6ab450443b3910c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695550a44a414e92b6ab450443b3910c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46321 } } }
I20260924 07:09:35.555899 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 695550a44a414e92b6ab450443b3910c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 695550a44a414e92b6ab450443b3910c. Latest consensus state: current_term: 1 leader_uuid: "695550a44a414e92b6ab450443b3910c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695550a44a414e92b6ab450443b3910c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46321 } } }
I20260924 07:09:35.555981 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 695550a44a414e92b6ab450443b3910c [sys.catalog]: This master's current role is: LEADER
I20260924 07:09:35.556000 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 695550a44a414e92b6ab450443b3910c [sys.catalog]: This master's current role is: LEADER
W20260924 07:09:35.556514 1018 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 695550a44a414e92b6ab450443b3910c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:09:35.556561 1018 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:09:35.556617 1019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:09:35.556792 1019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:09:35.557730 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:35.546772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34008 (local address 127.0.98.62:46321)
0924 07:09:35.546950 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:35.546959 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:35.546990 (+ 31us) server_negotiation.cc:407] Connection header received
0924 07:09:35.547486 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:35.547492 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:35.547535 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:35.547672 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:35.548264 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:35.549252 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:35.550050 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:35.550435 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:35.550547 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:35.553197 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:35.553201 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:35.553202 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:35.553214 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:35.555221 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:35.555666 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:35.555668 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:35.555669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:35.555786 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:35.556016 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:35.556017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:35.556017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:35.556412 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:35.556547 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:35.557406 (+ 859us) server_negotiation.cc:299] Negotiation successful
0924 07:09:35.557539 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
I20260924 07:09:35.558425 1019 catalog_manager.cc:1347] Generated new cluster ID: b856a843f5684955813210a62f2d3353
I20260924 07:09:35.558475 1019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:09:35.575860 1019 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:09:35.576351 1019 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:09:35.579267 1019 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 695550a44a414e92b6ab450443b3910c: Generated new TSK 0
I20260924 07:09:35.579427 1019 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 07:09:35.607776 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46321
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:35.709622 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:35.709817 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:35.709842 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:35.710981 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:35.711023 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:35.711050 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:09:35.712524 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46321
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:35.712738 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:35.712997 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:35.717828 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:35.717994 1024 server_base.cc:1034] running on GCE node
W20260924 07:09:35.717866 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:35.717818 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:35.718268 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:35.718509 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:35.719681 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790233775719673 us; error 31 us; skew 500 ppm
Sep 24 07:09:35 dist-test-slave-14ww krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:35.722075 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:09:35.722498 1024 webserver.cc:460] Webserver started at http://127.0.98.1:36853/ using document root <none> and password file <none>
I20260924 07:09:35.722707 1024 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:35.722750 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:35.722890 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:35.723886 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-0/data/instance:
uuid: "4a95abea39ca497ca0535f7d4534338c"
format_stamp: "Formatted at 2026-09-24 07:09:35 on dist-test-slave-14ww"
server_key: "db1ab890f66a209a7c3094652fe7d212"
server_key_iv: "2601c090d75bb9031df293ba51dfed04"
server_key_version: "encryptionkey@0"
I20260924 07:09:35.724236 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4a95abea39ca497ca0535f7d4534338c"
format_stamp: "Formatted at 2026-09-24 07:09:35 on dist-test-slave-14ww"
server_key: "db1ab890f66a209a7c3094652fe7d212"
server_key_iv: "2601c090d75bb9031df293ba51dfed04"
server_key_version: "encryptionkey@0"
I20260924 07:09:35.725550 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:09:35.726208 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:35.726361 1024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:09:35.726454 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-0/wal
uuid: "4a95abea39ca497ca0535f7d4534338c"
format_stamp: "Formatted at 2026-09-24 07:09:35 on dist-test-slave-14ww"
server_key: "db1ab890f66a209a7c3094652fe7d212"
server_key_iv: "2601c090d75bb9031df293ba51dfed04"
server_key_version: "encryptionkey@0"
I20260924 07:09:35.726523 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:35.746147 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:35.748123 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:35.748287 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:35.748517 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:35.748865 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:35.748917 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:35.748958 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:35.748986 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:35.754698 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39789
I20260924 07:09:35.754796 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39789 every 8 connection(s)
I20260924 07:09:35.755079 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 07:09:35 dist-test-slave-14ww krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:09:35.763016 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 1024
I20260924 07:09:35.763116 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:09:35.763335 392 external_mini_cluster.cc:1428] Setting key f13092badc400ab0561abe4f05cdf838
I20260924 07:09:35.764573 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:35.755771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46321 (local address 127.0.98.1:33761)
0924 07:09:35.755937 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:35.755969 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:35.756499 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:35.756700 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:35.756703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:35.757269 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:35.757451 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:35.757455 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:35.758186 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:35.758187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:35.758990 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:35.758992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:35.759383 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:35.759384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:35.759486 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:35.760046 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:35.760056 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:35.761481 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:35.763046 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:35.763052 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:35.763055 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:35.763357 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:35.763616 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:35.763617 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:35.763617 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:35.763770 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:35.764060 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:35.764066 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:35.764198 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 07:09:35.764305 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 07:09:35.764422 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":99,"threads_started":1}
I20260924 07:09:35.764586 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:35.755800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33761 (local address 127.0.98.62:46321)
0924 07:09:35.755868 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:35.755870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:35.756070 (+ 200us) server_negotiation.cc:407] Connection header received
0924 07:09:35.756582 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:35.756583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:35.756593 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:35.756649 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:35.757543 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:35.758075 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:35.759078 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:35.759278 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:35.759340 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:35.761578 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:35.761579 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:35.761580 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:35.761586 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:35.762948 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:35.763458 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:35.763459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:35.763459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:35.763526 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:35.763839 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:35.763840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:35.763840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:35.763987 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:35.764051 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:35.764350 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 07:09:35.764443 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 07:09:35.765059 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46321
I20260924 07:09:35.765158 1153 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:35.765342 1153 heartbeater.cc:511] Master 127.0.98.62:46321 requested a full tablet report, sending...
I20260924 07:09:35.765938 962 ts_manager.cc:194] Registered new tserver with Master: 4a95abea39ca497ca0535f7d4534338c (127.0.98.1:39789)
I20260924 07:09:35.766680 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33761
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 07:09:35.788297 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46321
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:35.870672 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:35.870857 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:35.870878 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:35.872037 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:35.872097 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:35.872141 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:09:35.873536 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46321
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:35.873720 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:35.873890 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:35.879019 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:35.879045 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:35.879133 1158 server_base.cc:1034] running on GCE node
W20260924 07:09:35.879078 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:35.879366 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:35.879595 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:35.880765 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790233775880720 us; error 27 us; skew 500 ppm
Sep 24 07:09:35 dist-test-slave-14ww krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:35.883209 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:09:35.883663 1158 webserver.cc:460] Webserver started at http://127.0.98.2:39737/ using document root <none> and password file <none>
I20260924 07:09:35.883956 1158 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:35.884025 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:35.884174 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:35.885228 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-1/data/instance:
uuid: "ea0abd008e0d4b96a364e3d2bae10217"
format_stamp: "Formatted at 2026-09-24 07:09:35 on dist-test-slave-14ww"
server_key: "72aa3257c1ed028c8e569ef630e2b383"
server_key_iv: "87748cf640ce4366e20419f7ba0536c4"
server_key_version: "encryptionkey@0"
I20260924 07:09:35.885565 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ea0abd008e0d4b96a364e3d2bae10217"
format_stamp: "Formatted at 2026-09-24 07:09:35 on dist-test-slave-14ww"
server_key: "72aa3257c1ed028c8e569ef630e2b383"
server_key_iv: "87748cf640ce4366e20419f7ba0536c4"
server_key_version: "encryptionkey@0"
I20260924 07:09:35.886842 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:09:35.887617 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:35.887843 1158 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:09:35.887916 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-1/wal
uuid: "ea0abd008e0d4b96a364e3d2bae10217"
format_stamp: "Formatted at 2026-09-24 07:09:35 on dist-test-slave-14ww"
server_key: "72aa3257c1ed028c8e569ef630e2b383"
server_key_iv: "87748cf640ce4366e20419f7ba0536c4"
server_key_version: "encryptionkey@0"
I20260924 07:09:35.887997 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:35.898226 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:35.900337 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:35.900585 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:35.900869 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:35.901216 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:35.901275 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:35.901316 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:35.901343 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:35.907552 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35969
I20260924 07:09:35.907588 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35969 every 8 connection(s)
I20260924 07:09:35.907896 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:09:35.912712 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 1158
I20260924 07:09:35.913554 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:09:35.913739 392 external_mini_cluster.cc:1428] Setting key 5880187debc728a6a47cb4dc1ac899a9
Sep 24 07:09:35 dist-test-slave-14ww krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:09:35.918054 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:35.908689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33079 (local address 127.0.98.62:46321)
0924 07:09:35.908786 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:35.908789 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:35.908950 (+ 161us) server_negotiation.cc:407] Connection header received
0924 07:09:35.909466 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:35.909468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:35.909478 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:35.909556 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:35.910531 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:35.911133 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:35.912158 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:35.912363 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:35.912420 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:35.914818 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:35.914820 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:35.914820 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:35.914828 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:35.916445 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:35.916943 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:35.916944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:35.916944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:35.917020 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:35.917356 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:35.917357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:35.917358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:35.917480 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:35.917538 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:35.917903 (+ 365us) server_negotiation.cc:299] Negotiation successful
0924 07:09:35.917982 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 07:09:35.918373 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:35.908636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46321 (local address 127.0.98.2:33079)
0924 07:09:35.908823 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:35.908865 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:35.909355 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:35.909632 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:35.909636 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:35.910214 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:35.910422 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:35.910427 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:35.911233 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:35.911235 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:35.912054 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:35.912056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:35.912447 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:35.912448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:35.912554 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:35.913132 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:35.913140 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:35.914715 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:35.916525 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:35.916532 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:35.916535 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:35.916864 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:35.917099 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:35.917100 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:35.917100 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:35.917241 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:35.917552 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:35.917558 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:35.917720 (+ 162us) client_negotiation.cc:769] Sending connection context
0924 07:09:35.917829 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 07:09:35.918011 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":125,"threads_started":1}
I20260924 07:09:35.918998 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46321
I20260924 07:09:35.919104 1287 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:35.919314 1287 heartbeater.cc:511] Master 127.0.98.62:46321 requested a full tablet report, sending...
I20260924 07:09:35.919675 962 ts_manager.cc:194] Registered new tserver with Master: ea0abd008e0d4b96a364e3d2bae10217 (127.0.98.2:35969)
I20260924 07:09:35.920054 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33079
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 07:09:36.767804 1153 heartbeater.cc:503] Master 127.0.98.62:46321 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260924 07:09:36.917029 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 1.002s user 0.000s sys 0.003s
I20260924 07:09:36.917379 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46321
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:09:36.921080 1287 heartbeater.cc:503] Master 127.0.98.62:46321 was elected leader, sending a full tablet report...
W20260924 07:09:37.001065 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:37.001300 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:37.001324 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:37.002522 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:37.002565 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:37.002591 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:09:37.003898 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46321
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:37.004066 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:37.004227 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:37.009189 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:37.009272 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:37.009270 1292 server_base.cc:1034] running on GCE node
W20260924 07:09:37.009198 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:37.009670 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:37.009866 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:37.011003 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790233777010988 us; error 29 us; skew 500 ppm
Sep 24 07:09:37 dist-test-slave-14ww krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:37.026002 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:09:37.026499 1292 webserver.cc:460] Webserver started at http://127.0.98.3:33643/ using document root <none> and password file <none>
I20260924 07:09:37.026729 1292 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:37.026795 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:37.026955 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:37.028116 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-2/data/instance:
uuid: "602bdcc957c54f69aa2d051bf9515380"
format_stamp: "Formatted at 2026-09-24 07:09:37 on dist-test-slave-14ww"
server_key: "30c84e3810e2df01a45722a11c5c0464"
server_key_iv: "09833caa9033a915beee1ba7537aa177"
server_key_version: "encryptionkey@0"
I20260924 07:09:37.028467 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-2/wal/instance:
uuid: "602bdcc957c54f69aa2d051bf9515380"
format_stamp: "Formatted at 2026-09-24 07:09:37 on dist-test-slave-14ww"
server_key: "30c84e3810e2df01a45722a11c5c0464"
server_key_iv: "09833caa9033a915beee1ba7537aa177"
server_key_version: "encryptionkey@0"
I20260924 07:09:37.029857 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:09:37.030639 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:37.030853 1292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:09:37.030952 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-2/wal
uuid: "602bdcc957c54f69aa2d051bf9515380"
format_stamp: "Formatted at 2026-09-24 07:09:37 on dist-test-slave-14ww"
server_key: "30c84e3810e2df01a45722a11c5c0464"
server_key_iv: "09833caa9033a915beee1ba7537aa177"
server_key_version: "encryptionkey@0"
I20260924 07:09:37.031054 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:37.048157 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:37.050140 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:37.050287 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:37.050493 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:37.050772 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:37.050827 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:37.050863 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:37.050897 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:37.057247 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43823
I20260924 07:09:37.057292 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43823 every 8 connection(s)
I20260924 07:09:37.057631 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:09:37.062314 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 1292
I20260924 07:09:37.062410 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790233772407728-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:09:37.062574 392 external_mini_cluster.cc:1428] Setting key 1ae264123ac8f52b8e7d088b36762e4e
Sep 24 07:09:37 dist-test-slave-14ww krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:09:37.067601 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:37.058252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53351 (local address 127.0.98.62:46321)
0924 07:09:37.058366 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:37.058368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:37.058484 (+ 116us) server_negotiation.cc:407] Connection header received
0924 07:09:37.058940 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:37.058940 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:37.058950 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:37.059012 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:37.059869 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:37.060838 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:37.061826 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:37.062174 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:37.062252 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:37.064616 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:37.064618 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:37.064618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:37.064626 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:37.066164 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:37.066614 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:37.066615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:37.066616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:37.066688 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:37.066954 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:37.066955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:37.066955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:37.067071 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:37.067154 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:37.067426 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 07:09:37.067508 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":71,"threads_started":1}
I20260924 07:09:37.067770 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:37.058222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46321 (local address 127.0.98.3:53351)
0924 07:09:37.058378 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:37.058418 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:37.058862 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:37.059072 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:37.059076 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:37.059624 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:37.059797 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:37.059801 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:37.060928 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:37.060929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:37.061731 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:37.061733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:37.062266 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:37.062267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:37.062358 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:37.062995 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:37.063004 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:37.064520 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:37.066277 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:37.066282 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:37.066285 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:37.066489 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:37.066771 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:37.066771 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:37.066772 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:37.066881 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:37.067145 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:37.067151 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:37.067297 (+ 146us) client_negotiation.cc:769] Sending connection context
0924 07:09:37.067404 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 07:09:37.067548 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":95,"threads_started":1}
I20260924 07:09:37.068267 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46321
I20260924 07:09:37.068346 1421 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:37.068529 1421 heartbeater.cc:511] Master 127.0.98.62:46321 requested a full tablet report, sending...
I20260924 07:09:37.068969 962 ts_manager.cc:194] Registered new tserver with Master: 602bdcc957c54f69aa2d051bf9515380 (127.0.98.3:43823)
I20260924 07:09:37.069375 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53351
I20260924 07:09:37.074260 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:09:37.081264 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:37.075810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34024 (local address 127.0.98.62:46321)
0924 07:09:37.075865 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:37.075867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:37.075986 (+ 119us) server_negotiation.cc:407] Connection header received
0924 07:09:37.076022 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:37.076022 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:37.076030 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:37.076076 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:37.076575 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:37.077082 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:37.077735 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:37.077930 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:37.077980 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:37.079086 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:37.079087 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:37.079087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:37.079092 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:37.080210 (+ 1118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:37.080498 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:37.080499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:37.080499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:37.080580 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:37.080785 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:37.080786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:37.080786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:37.080927 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:37.081003 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:37.081105 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 07:09:37.081161 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 07:09:37.083096 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:09:37.084606 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:09:37.091146 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:37.086850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39789 (local address 127.0.0.1:60074)
0924 07:09:37.087180 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:37.087216 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:37.087460 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:37.087709 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:37.087709 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:37.087711 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:37.087738 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:37.087738 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:37.088660 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:37.088661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:37.089966 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:37.089969 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:37.090890 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:37.090891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:37.090959 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:37.090963 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:09:37.091019 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 07:09:37.091059 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":29,"thread_start_us":76,"threads_started":1}
I20260924 07:09:37.091147 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:37.087026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60074 (local address 127.0.98.1:39789)
0924 07:09:37.087140 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:37.087144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:37.087281 (+ 137us) server_negotiation.cc:407] Connection header received
0924 07:09:37.087551 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:37.087553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:37.087568 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:37.087620 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:37.087832 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:37.088582 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:37.090053 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:37.090789 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:37.090877 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:37.090940 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:37.091033 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 07:09:37.091068 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":62,"threads_started":1}
I20260924 07:09:37.091166 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:37.087155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44762 (local address 127.0.98.2:35969)
0924 07:09:37.087271 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:37.087275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:37.087404 (+ 129us) server_negotiation.cc:407] Connection header received
0924 07:09:37.087473 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:37.087475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:37.087491 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:37.087566 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:37.087677 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:37.088915 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:37.089879 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:37.090601 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:37.090679 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:37.090732 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:37.091046 (+ 314us) server_negotiation.cc:299] Negotiation successful
0924 07:09:37.091093 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":53,"threads_started":1}
I20260924 07:09:37.091003 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:37.087062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35969 (local address 127.0.0.1:44762)
0924 07:09:37.087330 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:37.087339 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:37.087396 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:37.087566 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:37.087566 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:37.087568 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:37.087620 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:37.087621 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:37.088977 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:37.088978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:37.089779 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:37.089781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:37.090690 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:37.090692 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:37.090779 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:37.090792 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 07:09:37.090857 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 07:09:37.090907 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":1,"thread_start_us":63,"threads_started":1}
I20260924 07:09:37.091490 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:37.086933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52946 (local address 127.0.98.3:43823)
0924 07:09:37.087333 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:37.087338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:37.087348 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:09:37.087385 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:37.087386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:37.087402 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:37.087478 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:37.087656 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:37.088817 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:37.090069 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:37.090708 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:37.090773 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:37.090824 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:37.091098 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 07:09:37.091135 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":84,"threads_started":1}
I20260924 07:09:37.091827 1087 tablet_service.cc:1467] Processing CreateTablet for tablet e0e0db479aca48998d50a4cefa474c8c (DEFAULT_TABLE table=test-table [id=67e1bd2166cc4191bda5b94cab19da71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:37.091924 1221 tablet_service.cc:1467] Processing CreateTablet for tablet e0e0db479aca48998d50a4cefa474c8c (DEFAULT_TABLE table=test-table [id=67e1bd2166cc4191bda5b94cab19da71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:37.092255 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0e0db479aca48998d50a4cefa474c8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:37.091068 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:37.086849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43823 (local address 127.0.0.1:52946)
0924 07:09:37.087178 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:37.087213 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:37.087281 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:37.087491 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:37.087494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:37.087496 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:37.087564 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:37.087565 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:37.088912 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:37.088913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:37.089984 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:37.089985 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:37.090855 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:37.090855 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:37.090915 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:37.090918 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:09:37.090971 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 07:09:37.091001 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":1,"thread_start_us":83,"threads_started":1}
I20260924 07:09:37.092163 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0e0db479aca48998d50a4cefa474c8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:37.094820 1440 tablet_bootstrap.cc:492] T e0e0db479aca48998d50a4cefa474c8c P ea0abd008e0d4b96a364e3d2bae10217: Bootstrap starting.
I20260924 07:09:37.095489 1441 tablet_bootstrap.cc:492] T e0e0db479aca48998d50a4cefa474c8c P 4a95abea39ca497ca0535f7d4534338c: Bootstrap starting.
I20260924 07:09:37.095646 1440 tablet_bootstrap.cc:654] T e0e0db479aca48998d50a4cefa474c8c P ea0abd008e0d4b96a364e3d2bae10217: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:37.095551 1355 tablet_service.cc:1467] Processing CreateTablet for tablet e0e0db479aca48998d50a4cefa474c8c (DEFAULT_TABLE table=test-table [id=67e1bd2166cc4191bda5b94cab19da71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:37.095908 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0e0db479aca48998d50a4cefa474c8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:37.095964 1440 log.cc:824] T e0e0db479aca48998d50a4cefa474c8c P ea0abd008e0d4b96a364e3d2bae10217: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:37.096149 1441 tablet_bootstrap.cc:654] T e0e0db479aca48998d50a4cefa474c8c P 4a95abea39ca497ca0535f7d4534338c: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:37.096483 1441 log.cc:824] T e0e0db479aca48998d50a4cefa474c8c P 4a95abea39ca497ca0535f7d4534338c: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:37.096722 1440 tablet_bootstrap.cc:492] T e0e0db479aca48998d50a4cefa474c8c P ea0abd008e0d4b96a364e3d2bae10217: No bootstrap required, opened a new log
I20260924 07:09:37.096822 1440 ts_tablet_manager.cc:1397] T e0e0db479aca48998d50a4cefa474c8c P ea0abd008e0d4b96a364e3d2bae10217: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:09:37.097129 1441 tablet_bootstrap.cc:492] T e0e0db479aca48998d50a4cefa474c8c P 4a95abea39ca497ca0535f7d4534338c: No bootstrap required, opened a new log
I20260924 07:09:37.097208 1441 ts_tablet_manager.cc:1397] T e0e0db479aca48998d50a4cefa474c8c P 4a95abea39ca497ca0535f7d4534338c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:09:37.099124 1441 raft_consensus.cc:348] T e0e0db479aca48998d50a4cefa474c8c P 4a95abea39ca497ca0535f7d4534338c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "602bdcc957c54f69aa2d051bf9515380" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43823 } } peers { permanent_uuid: "ea0abd008e0d4b96a364e3d2bae10217" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35969 } } peers { permanent_uuid: "4a95abea39ca497ca0535f7d4534338c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39789 } }
I20260924 07:09:37.099277 1441 raft_consensus.cc:374] T e0e0db479aca48998d50a4cefa474c8c P 4a95abea39ca497ca0535f7d4534338c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:37.099321 1441 raft_consensus.cc:729] T e0e0db479aca48998d50a4cefa474c8c P 4a95abea39ca497ca0535f7d4534338c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a95abea39ca497ca0535f7d4534338c, State: Initialized, Role: FOLLOWER
I20260924 07:09:37.099460 1441 consensus_queue.cc:260] T e0e0db479aca48998d50a4cefa474c8c P 4a95abea39ca497ca0535f7d4534338c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "602bdcc957c54f69aa2d051bf9515380" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43823 } } peers { permanent_uuid: "ea0abd008e0d4b96a364e3d2bae10217" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35969 } } peers { permanent_uuid: "4a95abea39ca497ca0535f7d4534338c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39789 } }
I20260924 07:09:37.099792 1441 ts_tablet_manager.cc:1428] T e0e0db479aca48998d50a4cefa474c8c P 4a95abea39ca497ca0535f7d4534338c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 07:09:37.099757 1440 raft_consensus.cc:348] T e0e0db479aca48998d50a4cefa474c8c P ea0abd008e0d4b96a364e3d2bae10217 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "602bdcc957c54f69aa2d051bf9515380" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43823 } } peers { permanent_uuid: "ea0abd008e0d4b96a364e3d2bae10217" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35969 } } peers { permanent_uuid: "4a95abea39ca497ca0535f7d4534338c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39789 } }
I20260924 07:09:37.099929 1440 raft_consensus.cc:374] T e0e0db479aca48998d50a4cefa474c8c P ea0abd008e0d4b96a364e3d2bae10217 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:37.099962 1440 raft_consensus.cc:729] T e0e0db479aca48998d50a4cefa474c8c P ea0abd008e0d4b96a364e3d2bae10217 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea0abd008e0d4b96a364e3d2bae10217, State: Initialized, Role: FOLLOWER
I20260924 07:09:37.100069 1440 consensus_queue.cc:260] T e0e0db479aca48998d50a4cefa474c8c P ea0abd008e0d4b96a364e3d2bae10217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "602bdcc957c54f69aa2d051bf9515380" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43823 } } peers { permanent_uuid: "ea0abd008e0d4b96a364e3d2bae10217" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35969 } } peers { permanent_uuid: "4a95abea39ca497ca0535f7d4534338c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39789 } }
I20260924 07:09:37.100343 1440 ts_tablet_manager.cc:1428] T e0e0db479aca48998d50a4cefa474c8c P ea0abd008e0d4b96a364e3d2bae10217: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 07:09:37.100512 1444 tablet_bootstrap.cc:492] T e0e0db479aca48998d50a4cefa474c8c P 602bdcc957c54f69aa2d051bf9515380: Bootstrap starting.
I20260924 07:09:37.101281 1444 tablet_bootstrap.cc:654] T e0e0db479aca48998d50a4cefa474c8c P 602bdcc957c54f69aa2d051bf9515380: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:37.101737 1444 log.cc:824] T e0e0db479aca48998d50a4cefa474c8c P 602bdcc957c54f69aa2d051bf9515380: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:37.102341 1444 tablet_bootstrap.cc:492] T e0e0db479aca48998d50a4cefa474c8c P 602bdcc957c54f69aa2d051bf9515380: No bootstrap required, opened a new log
I20260924 07:09:37.102391 1444 ts_tablet_manager.cc:1397] T e0e0db479aca48998d50a4cefa474c8c P 602bdcc957c54f69aa2d051bf9515380: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 07:09:37.103750 1444 raft_consensus.cc:348] T e0e0db479aca48998d50a4cefa474c8c P 602bdcc957c54f69aa2d051bf9515380 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "602bdcc957c54f69aa2d051bf9515380" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43823 } } peers { permanent_uuid: "ea0abd008e0d4b96a364e3d2bae10217" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35969 } } peers { permanent_uuid: "4a95abea39ca497ca0535f7d4534338c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39789 } }
I20260924 07:09:37.103871 1444 raft_consensus.cc:374] T e0e0db479aca48998d50a4cefa474c8c P 602bdcc957c54f69aa2d051bf9515380 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:37.103892 1444 raft_consensus.cc:729] T e0e0db479aca48998d50a4cefa474c8c P 602bdcc957c54f69aa2d051bf9515380 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 602bdcc957c54f69aa2d051bf9515380, State: Initialized, Role: FOLLOWER
I20260924 07:09:37.104017 1444 consensus_queue.cc:260] T e0e0db479aca48998d50a4cefa474c8c P 602bdcc957c54f69aa2d051bf9515380 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "602bdcc957c54f69aa2d051bf9515380" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43823 } } peers { permanent_uuid: "ea0abd008e0d4b96a364e3d2bae10217" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35969 } } peers { permanent_uuid: "4a95abea39ca497ca0535f7d4534338c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39789 } }
I20260924 07:09:37.104264 1444 ts_tablet_manager.cc:1428] T e0e0db479aca48998d50a4cefa474c8c P 602bdcc957c54f69aa2d051bf9515380: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 07:09:37.104286 1421 heartbeater.cc:503] Master 127.0.98.62:46321 was elected leader, sending a full tablet report...
I20260924 07:09:37.130342 1445 raft_consensus.cc:482] T e0e0db479aca48998d50a4cefa474c8c P 4a95abea39ca497ca0535f7d4534338c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:09:37.130434 1445 raft_consensus.cc:504] T e0e0db479aca48998d50a4cefa474c8c P 4a95abea39ca497ca0535f7d4534338c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "602bdcc957c54f69aa2d051bf9515380" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43823 } } peers { permanent_uuid: "ea0abd008e0d4b96a364e3d2bae10217" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35969 } } peers { permanent_uuid: "4a95abea39ca497ca0535f7d4534338c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39789 } }
I20260924 07:09:37.130779 1445 leader_election.cc:288] T e0e0db479aca48998d50a4cefa474c8c P 4a95abea39ca497ca0535f7d4534338c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 602bdcc957c54f69aa2d051bf9515380 (127.0.98.3:43823), ea0abd008e0d4b96a364e3d2bae10217 (127.0.98.2:35969)
I20260924 07:09:37.135299 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:37.130939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43823 (local address 127.0.98.1:36589)
0924 07:09:37.131117 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:37.131133 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:37.131211 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:37.131426 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:37.131427 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:37.131428 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:37.131508 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:37.131510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:37.132375 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:37.132376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:37.134169 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:37.134172 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:37.135005 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:37.135007 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:37.135087 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:37.135093 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 07:09:37.135159 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 07:09:37.135208 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":23,"thread_start_us":95,"threads_started":1}
I20260924 07:09:37.135331 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:37.131155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35969 (local address 127.0.98.1:40163)
0924 07:09:37.131267 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:37.131276 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:37.131333 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:37.131454 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:37.131454 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:37.131455 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:37.131520 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:37.131521 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:37.132387 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:37.132388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:37.134203 (+ 1815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:37.134273 (+ 70us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:37.135007 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:37.135008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:37.135087 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:37.135091 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:09:37.135159 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 07:09:37.135250 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":85,"thread_start_us":62,"threads_started":1}
I20260924 07:09:37.135329 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:37.131004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36589 (local address 127.0.98.3:43823)
0924 07:09:37.131071 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:37.131074 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:37.131194 (+ 120us) server_negotiation.cc:407] Connection header received
0924 07:09:37.131323 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:37.131324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:37.131337 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:37.131398 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:37.131570 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:37.132291 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:37.134337 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:37.134922 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:37.134992 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:37.135001 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:37.135204 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 07:09:37.135243 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 07:09:37.135345 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:37.131228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40163 (local address 127.0.98.2:35969)
0924 07:09:37.131273 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:37.131275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:37.131323 (+ 48us) server_negotiation.cc:407] Connection header received
0924 07:09:37.131395 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:37.131396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:37.131405 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:37.131454 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:37.131571 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:37.132297 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:37.134354 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:37.134920 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:37.134992 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:37.135001 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:37.135199 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 07:09:37.135243 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 07:09:37.135712 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0e0db479aca48998d50a4cefa474c8c" candidate_uuid: "4a95abea39ca497ca0535f7d4534338c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "602bdcc957c54f69aa2d051bf9515380" is_pre_election: true
I20260924 07:09:37.135757 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0e0db479aca48998d50a4cefa474c8c" candidate_uuid: "4a95abea39ca497ca0535f7d4534338c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea0abd008e0d4b96a364e3d2bae10217" is_pre_election: true
I20260924 07:09:37.135891 1241 raft_consensus.cc:2454] T e0e0db479aca48998d50a4cefa474c8c P ea0abd008e0d4b96a364e3d2bae10217 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a95abea39ca497ca0535f7d4534338c in term 0.
I20260924 07:09:37.135891 1375 raft_consensus.cc:2454] T e0e0db479aca48998d50a4cefa474c8c P 602bdcc957c54f69aa2d051bf9515380 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a95abea39ca497ca0535f7d4534338c in term 0.
I20260924 07:09:37.136106 1043 leader_election.cc:302] T e0e0db479aca48998d50a4cefa474c8c P 4a95abea39ca497ca0535f7d4534338c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a95abea39ca497ca0535f7d4534338c, 602bdcc957c54f69aa2d051bf9515380; no voters:
I20260924 07:09:37.136233 1445 raft_consensus.cc:2781] T e0e0db479aca48998d50a4cefa474c8c P 4a95abea39ca497ca0535f7d4534338c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:09:37.136288 1445 raft_consensus.cc:482] T e0e0db479aca48998d50a4cefa474c8c P 4a95abea39ca497ca0535f7d4534338c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:09:37.136317 1445 raft_consensus.cc:3037] T e0e0db479aca48998d50a4cefa474c8c P 4a95abea39ca497ca0535f7d4534338c [term 0 FOLLOWER]: Advancing to term 1
W20260924 07:09:37.159085 1288 tablet.cc:2369] T e0e0db479aca48998d50a4cefa474c8c P ea0abd008e0d4b96a364e3d2bae10217: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:09:37.204841 1445 raft_consensus.cc:504] T e0e0db479aca48998d50a4cefa474c8c P 4a95abea39ca497ca0535f7d4534338c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "602bdcc957c54f69aa2d051bf9515380" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43823 } } peers { permanent_uuid: "ea0abd008e0d4b96a364e3d2bae10217" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35969 } } peers { permanent_uuid: "4a95abea39ca497ca0535f7d4534338c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39789 } }
I20260924 07:09:37.205057 1445 leader_election.cc:288] T e0e0db479aca48998d50a4cefa474c8c P 4a95abea39ca497ca0535f7d4534338c [CANDIDATE]: Term 1 election: Requested vote from peers 602bdcc957c54f69aa2d051bf9515380 (127.0.98.3:43823), ea0abd008e0d4b96a364e3d2bae10217 (127.0.98.2:35969)
I20260924 07:09:37.205291 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0e0db479aca48998d50a4cefa474c8c" candidate_uuid: "4a95abea39ca497ca0535f7d4534338c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "602bdcc957c54f69aa2d051bf9515380"
I20260924 07:09:37.205286 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0e0db479aca48998d50a4cefa474c8c" candidate_uuid: "4a95abea39ca497ca0535f7d4534338c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea0abd008e0d4b96a364e3d2bae10217"
I20260924 07:09:37.205377 1375 raft_consensus.cc:3037] T e0e0db479aca48998d50a4cefa474c8c P 602bdcc957c54f69aa2d051bf9515380 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:37.205397 1241 raft_consensus.cc:3037] T e0e0db479aca48998d50a4cefa474c8c P ea0abd008e0d4b96a364e3d2bae10217 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:37.206254 1375 raft_consensus.cc:2454] T e0e0db479aca48998d50a4cefa474c8c P 602bdcc957c54f69aa2d051bf9515380 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a95abea39ca497ca0535f7d4534338c in term 1.
I20260924 07:09:37.206468 1043 leader_election.cc:302] T e0e0db479aca48998d50a4cefa474c8c P 4a95abea39ca497ca0535f7d4534338c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a95abea39ca497ca0535f7d4534338c, 602bdcc957c54f69aa2d051bf9515380; no voters:
I20260924 07:09:37.206564 1241 raft_consensus.cc:2454] T e0e0db479aca48998d50a4cefa474c8c P ea0abd008e0d4b96a364e3d2bae10217 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a95abea39ca497ca0535f7d4534338c in term 1.
I20260924 07:09:37.206591 1445 raft_consensus.cc:2781] T e0e0db479aca48998d50a4cefa474c8c P 4a95abea39ca497ca0535f7d4534338c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:09:37.206751 1445 raft_consensus.cc:686] T e0e0db479aca48998d50a4cefa474c8c P 4a95abea39ca497ca0535f7d4534338c [term 1 LEADER]: Becoming Leader. State: Replica: 4a95abea39ca497ca0535f7d4534338c, State: Running, Role: LEADER
I20260924 07:09:37.206856 1445 consensus_queue.cc:237] T e0e0db479aca48998d50a4cefa474c8c P 4a95abea39ca497ca0535f7d4534338c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "602bdcc957c54f69aa2d051bf9515380" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43823 } } peers { permanent_uuid: "ea0abd008e0d4b96a364e3d2bae10217" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35969 } } peers { permanent_uuid: "4a95abea39ca497ca0535f7d4534338c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39789 } }
I20260924 07:09:37.207783 961 catalog_manager.cc:5495] T e0e0db479aca48998d50a4cefa474c8c P 4a95abea39ca497ca0535f7d4534338c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a95abea39ca497ca0535f7d4534338c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "4a95abea39ca497ca0535f7d4534338c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "602bdcc957c54f69aa2d051bf9515380" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea0abd008e0d4b96a364e3d2bae10217" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a95abea39ca497ca0535f7d4534338c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39789 } health_report { overall_health: HEALTHY } } }
Sep 24 07:09:37 dist-test-slave-14ww krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 07:09:37.239446 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:37.231367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60090 (local address 127.0.98.1:39789)
0924 07:09:37.231433 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:37.231435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:37.231501 (+ 66us) server_negotiation.cc:407] Connection header received
0924 07:09:37.231544 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:37.231545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:37.231557 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:37.231611 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:37.232310 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:37.232935 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:37.233877 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:37.234056 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:37.234108 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:37.236495 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:37.236504 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:37.236504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:37.236511 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:37.238330 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:37.238648 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:37.238650 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:37.238651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:37.238708 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:37.238877 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:37.238878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:37.238878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:37.239016 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:37.239118 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:37.239205 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 07:09:37.239268 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Sep 24 07:09:37 dist-test-slave-14ww krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233777, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260924 07:09:37.256402 1154 tablet.cc:2369] T e0e0db479aca48998d50a4cefa474c8c P 4a95abea39ca497ca0535f7d4534338c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 07:09:37 dist-test-slave-14ww krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233777, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
W20260924 07:09:37.308133 1422 tablet.cc:2369] T e0e0db479aca48998d50a4cefa474c8c P 602bdcc957c54f69aa2d051bf9515380: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:09:37.309816 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:37.302198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60104 (local address 127.0.98.1:39789)
0924 07:09:37.302294 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:37.302297 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:37.302377 (+ 80us) server_negotiation.cc:407] Connection header received
0924 07:09:37.302417 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:37.302418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:37.302431 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:37.302493 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:37.303075 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:37.303904 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:37.304658 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:37.304871 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:37.304931 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:37.307204 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:37.307206 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:37.307206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:37.307213 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:37.308726 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:37.309103 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:37.309104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:37.309104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:37.309154 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:37.309323 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:37.309324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:37.309324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:37.309418 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:37.309505 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:37.309680 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 07:09:37.309741 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Sep 24 07:09:37 dist-test-slave-14ww krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 07:09:37 dist-test-slave-14ww krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 07:09:37.410372 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:37.402580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60108 (local address 127.0.98.1:39789)
0924 07:09:37.402651 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:37.402653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:37.402745 (+ 92us) server_negotiation.cc:407] Connection header received
0924 07:09:37.402850 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:37.402851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:37.402864 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:37.402943 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:37.403544 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:37.404032 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:37.404824 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:37.405025 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:37.405103 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:37.407354 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:37.407356 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:37.407356 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:37.407364 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:37.408777 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:37.409593 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:37.409594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:37.409594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:37.409655 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:37.409867 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:37.409868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:37.409868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:37.409976 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:37.410104 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:37.410187 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 07:09:37.410264 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260924 07:09:37.410876 1067 tablet_service.cc:2908] Rejecting scan request for tablet e0e0db479aca48998d50a4cefa474c8c: Uninitialized: clean time has not yet been initialized
I20260924 07:09:37.411821 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 1024
I20260924 07:09:37.417270 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 1158
I20260924 07:09:37.421654 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 1292
I20260924 07:09:37.426011 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 930
2026-09-24T07:09:37Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3156 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:09:38 dist-test-slave-14ww krb5kdc[1465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:09:38 dist-test-slave-14ww krb5kdc[1465](info): set up 2 sockets
Sep 24 07:09:38 dist-test-slave-14ww krb5kdc[1465](info): commencing operation
krb5kdc: starting...
W20260924 07:09:39.919854 392 mini_kdc.cc:121] Time spent starting KDC: real 2.472s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260924 07:09:40.110991 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.191s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260924 07:09:40.304457 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.193s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260924 07:09:40.413900 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.109s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/krb5kdc/test-user.keytab.
W20260924 07:09:40.856034 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.442s user 0.000s sys 0.002s
Sep 24 07:09:40 dist-test-slave-14ww krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:09:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:09:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260924 07:09:41.687386 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.810s user 0.000s sys 0.002s
I20260924 07:09:41.687713 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35085
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:41.765620 1478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:41.765802 1478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:41.765825 1478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:41.766970 1478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:09:41.767010 1478 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 07:09:41.767024 1478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:41.767037 1478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:41.767050 1478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:09:41.767061 1478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:09:41.768533 1478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35085
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:41.768783 1478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:41.769044 1478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:41.773716 1483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:41.773694 1486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:41.773880 1484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:41.773859 1478 server_base.cc:1034] running on GCE node
I20260924 07:09:41.774191 1478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:41.774432 1478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:41.775599 1478 hybrid_clock.cc:648] HybridClock initialized: now 1790233781775575 us; error 37 us; skew 500 ppm
Sep 24 07:09:41 dist-test-slave-14ww krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233781, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:41.778328 1478 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:09:41.778838 1478 webserver.cc:460] Webserver started at http://127.0.98.62:33033/ using document root <none> and password file <none>
I20260924 07:09:41.779102 1478 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:41.779181 1478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:41.779340 1478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:41.780452 1478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "5528fa3ab4b145ebbc6cd8f9eb434d15"
format_stamp: "Formatted at 2026-09-24 07:09:41 on dist-test-slave-14ww"
server_key: "d060569e02e20b9f1767a9ae945edf56"
server_key_iv: "2b0d63bf82094a19c1ab9642b4096039"
server_key_version: "encryptionkey@0"
I20260924 07:09:41.780910 1478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "5528fa3ab4b145ebbc6cd8f9eb434d15"
format_stamp: "Formatted at 2026-09-24 07:09:41 on dist-test-slave-14ww"
server_key: "d060569e02e20b9f1767a9ae945edf56"
server_key_iv: "2b0d63bf82094a19c1ab9642b4096039"
server_key_version: "encryptionkey@0"
I20260924 07:09:41.782328 1478 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:09:41.783056 1493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:41.783227 1478 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:41.783321 1478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "5528fa3ab4b145ebbc6cd8f9eb434d15"
format_stamp: "Formatted at 2026-09-24 07:09:41 on dist-test-slave-14ww"
server_key: "d060569e02e20b9f1767a9ae945edf56"
server_key_iv: "2b0d63bf82094a19c1ab9642b4096039"
server_key_version: "encryptionkey@0"
I20260924 07:09:41.783401 1478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:41.793305 1478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:41.795097 1478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:41.795258 1478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:41.799837 1478 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35085
I20260924 07:09:41.799882 1555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35085 every 8 connection(s)
I20260924 07:09:41.800184 1478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:09:41.800801 1556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:41.802011 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 1478
I20260924 07:09:41.802089 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:09:41.802249 392 external_mini_cluster.cc:1428] Setting key fa4a7cb428c821b53d4d8384be74f57c
I20260924 07:09:41.803447 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5528fa3ab4b145ebbc6cd8f9eb434d15: Bootstrap starting.
I20260924 07:09:41.804119 1556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5528fa3ab4b145ebbc6cd8f9eb434d15: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:41.804427 1556 log.cc:824] T 00000000000000000000000000000000 P 5528fa3ab4b145ebbc6cd8f9eb434d15: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:41.805213 1556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5528fa3ab4b145ebbc6cd8f9eb434d15: No bootstrap required, opened a new log
I20260924 07:09:41.807458 1556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5528fa3ab4b145ebbc6cd8f9eb434d15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5528fa3ab4b145ebbc6cd8f9eb434d15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35085 } }
I20260924 07:09:41.807655 1556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5528fa3ab4b145ebbc6cd8f9eb434d15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:41.807694 1556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5528fa3ab4b145ebbc6cd8f9eb434d15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5528fa3ab4b145ebbc6cd8f9eb434d15, State: Initialized, Role: FOLLOWER
I20260924 07:09:41.807814 1556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5528fa3ab4b145ebbc6cd8f9eb434d15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5528fa3ab4b145ebbc6cd8f9eb434d15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35085 } }
I20260924 07:09:41.807909 1556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5528fa3ab4b145ebbc6cd8f9eb434d15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:09:41.807945 1556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5528fa3ab4b145ebbc6cd8f9eb434d15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:09:41.808002 1556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5528fa3ab4b145ebbc6cd8f9eb434d15 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:41.808506 1556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5528fa3ab4b145ebbc6cd8f9eb434d15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5528fa3ab4b145ebbc6cd8f9eb434d15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35085 } }
I20260924 07:09:41.808589 1556 leader_election.cc:302] T 00000000000000000000000000000000 P 5528fa3ab4b145ebbc6cd8f9eb434d15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5528fa3ab4b145ebbc6cd8f9eb434d15; no voters:
I20260924 07:09:41.808789 1556 leader_election.cc:288] T 00000000000000000000000000000000 P 5528fa3ab4b145ebbc6cd8f9eb434d15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:09:41.808818 1561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5528fa3ab4b145ebbc6cd8f9eb434d15 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:09:41.808924 1561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5528fa3ab4b145ebbc6cd8f9eb434d15 [term 1 LEADER]: Becoming Leader. State: Replica: 5528fa3ab4b145ebbc6cd8f9eb434d15, State: Running, Role: LEADER
I20260924 07:09:41.809015 1561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5528fa3ab4b145ebbc6cd8f9eb434d15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5528fa3ab4b145ebbc6cd8f9eb434d15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35085 } }
I20260924 07:09:41.809049 1556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5528fa3ab4b145ebbc6cd8f9eb434d15 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:09:41.809327 1561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5528fa3ab4b145ebbc6cd8f9eb434d15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5528fa3ab4b145ebbc6cd8f9eb434d15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5528fa3ab4b145ebbc6cd8f9eb434d15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35085 } } }
I20260924 07:09:41.809407 1561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5528fa3ab4b145ebbc6cd8f9eb434d15 [sys.catalog]: This master's current role is: LEADER
I20260924 07:09:41.809556 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5528fa3ab4b145ebbc6cd8f9eb434d15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5528fa3ab4b145ebbc6cd8f9eb434d15. Latest consensus state: current_term: 1 leader_uuid: "5528fa3ab4b145ebbc6cd8f9eb434d15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5528fa3ab4b145ebbc6cd8f9eb434d15" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35085 } } }
Sep 24 07:09:41 dist-test-slave-14ww krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:09:41.809620 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5528fa3ab4b145ebbc6cd8f9eb434d15 [sys.catalog]: This master's current role is: LEADER
I20260924 07:09:41.809710 1566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:09:41.810252 1566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:09:41.812080 1566 catalog_manager.cc:1347] Generated new cluster ID: dbccf9abae024a608c601ec0a8aa0e3a
I20260924 07:09:41.812142 1566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:09:41.814101 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:41.802745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36156 (local address 127.0.98.62:35085)
0924 07:09:41.802935 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:41.802945 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:41.802978 (+ 33us) server_negotiation.cc:407] Connection header received
0924 07:09:41.803512 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:41.803520 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:41.803580 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:41.803736 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:41.804524 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:41.805943 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:41.806700 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:41.807304 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:41.807427 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:41.810228 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:41.810232 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:41.810233 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:41.810245 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:41.812556 (+ 2311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:41.813037 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:41.813040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:41.813040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:41.813128 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:41.813337 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:41.813338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:41.813338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:41.813566 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:41.813748 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:41.813813 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 07:09:41.813941 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":83,"threads_started":1}
I20260924 07:09:41.827298 1566 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:09:41.828004 1566 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:09:41.836912 1566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5528fa3ab4b145ebbc6cd8f9eb434d15: Generated new TSK 0
I20260924 07:09:41.837114 1566 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 07:09:41.867254 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35085
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:41.973378 1582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:41.973593 1582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:41.973634 1582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:41.974776 1582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:41.974840 1582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:41.974875 1582 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:09:41.974927 1582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:09:41.976389 1582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:41.976646 1582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:41.976890 1582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:41.981870 1587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:41.981894 1590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:41.981870 1588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:41.982371 1582 server_base.cc:1034] running on GCE node
I20260924 07:09:41.982589 1582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:41.982793 1582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:41.983930 1582 hybrid_clock.cc:648] HybridClock initialized: now 1790233781983919 us; error 30 us; skew 500 ppm
Sep 24 07:09:41 dist-test-slave-14ww krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233781, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:41.986688 1582 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:09:41.987094 1582 webserver.cc:460] Webserver started at http://127.0.98.1:36059/ using document root <none> and password file <none>
I20260924 07:09:41.987320 1582 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:41.987365 1582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:41.987461 1582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:41.988567 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-0/data/instance:
uuid: "f1ab124407d1401187fcc2295578d2ad"
format_stamp: "Formatted at 2026-09-24 07:09:41 on dist-test-slave-14ww"
server_key: "c1ebb4a363e6494e6a9a67089d0d3453"
server_key_iv: "c38a7c21bdbc77b3729b228a89ba3e11"
server_key_version: "encryptionkey@0"
I20260924 07:09:41.988958 1582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f1ab124407d1401187fcc2295578d2ad"
format_stamp: "Formatted at 2026-09-24 07:09:41 on dist-test-slave-14ww"
server_key: "c1ebb4a363e6494e6a9a67089d0d3453"
server_key_iv: "c38a7c21bdbc77b3729b228a89ba3e11"
server_key_version: "encryptionkey@0"
I20260924 07:09:41.990320 1582 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:09:41.991040 1597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:41.991211 1582 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:09:41.991291 1582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-0/wal
uuid: "f1ab124407d1401187fcc2295578d2ad"
format_stamp: "Formatted at 2026-09-24 07:09:41 on dist-test-slave-14ww"
server_key: "c1ebb4a363e6494e6a9a67089d0d3453"
server_key_iv: "c38a7c21bdbc77b3729b228a89ba3e11"
server_key_version: "encryptionkey@0"
I20260924 07:09:41.991343 1582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:42.009804 1582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:42.012075 1582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:42.012187 1582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:42.012627 1582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:42.012661 1582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:42.012684 1582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:42.012697 1582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:42.019464 1582 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46873
I20260924 07:09:42.019541 1712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46873 every 8 connection(s)
I20260924 07:09:42.019847 1582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 07:09:42 dist-test-slave-14ww krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233781, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:09:42.022745 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 1582
I20260924 07:09:42.022866 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:09:42.023043 392 external_mini_cluster.cc:1428] Setting key ebc19e8949cc636440b04d22b7271e79
I20260924 07:09:42.024051 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.013287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40731 (local address 127.0.98.62:35085)
0924 07:09:42.013359 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:42.013362 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:42.013570 (+ 208us) server_negotiation.cc:407] Connection header received
0924 07:09:42.014409 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:42.014411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:42.014425 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:42.014509 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:42.015463 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.016419 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.018111 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.018527 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.018617 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:42.021204 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:42.021206 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:42.021206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:42.021213 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:42.022563 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:42.022974 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:42.022975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:42.022975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:42.023025 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:42.023367 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:42.023368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:42.023368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:42.023496 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:42.023581 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:42.023911 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 07:09:42.023989 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 07:09:42.024413 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.013270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35085 (local address 127.0.98.1:40731)
0924 07:09:42.013445 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:42.013486 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:42.014215 (+ 729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:42.014648 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:42.014653 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:42.015056 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:42.015351 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:42.015356 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.016543 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.016544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:42.018019 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:42.018021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.018634 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.018636 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:42.018818 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:42.019496 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:42.019506 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:42.021117 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:42.022646 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:42.022652 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:42.022653 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:42.022872 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:42.023096 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:42.023098 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:42.023098 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:42.023269 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:42.023570 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:42.023577 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:42.023760 (+ 183us) client_negotiation.cc:769] Sending connection context
0924 07:09:42.023894 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 07:09:42.024089 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":101,"threads_started":1}
I20260924 07:09:42.025172 1713 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35085
I20260924 07:09:42.025301 1713 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:42.025489 1713 heartbeater.cc:511] Master 127.0.98.62:35085 requested a full tablet report, sending...
I20260924 07:09:42.026001 1510 ts_manager.cc:194] Registered new tserver with Master: f1ab124407d1401187fcc2295578d2ad (127.0.98.1:46873)
I20260924 07:09:42.026929 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40731
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 07:09:42.046185 1720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.026125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35085 (local address 127.0.98.1:41765)
0924 07:09:42.026283 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:42.026294 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:42.026345 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:42.026475 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:42.026476 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:42.026660 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:42.026715 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:42.026716 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.027575 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.027576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:42.028501 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:42.028503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.028947 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.028949 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:42.029021 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:42.043173 (+ 14152us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:42.043178 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:42.043478 (+ 300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:42.045128 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:42.045129 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:42.045129 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:42.045294 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:42.045536 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:42.045536 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:42.045537 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:42.045590 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:42.045925 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:42.045926 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:42.045967 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 07:09:42.046024 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 07:09:42.046107 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":88,"threads_started":1}
I20260924 07:09:42.046209 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.026305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41765 (local address 127.0.98.62:35085)
0924 07:09:42.026345 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:42.026347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:42.026356 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:09:42.026407 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:42.026408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:42.026416 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:42.026460 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:42.026778 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.027467 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.028594 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.028872 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.028934 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:42.043582 (+ 14648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:42.043584 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:42.043585 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:42.043592 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:42.045044 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:42.045392 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:42.045393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:42.045393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:42.045465 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:42.045651 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:42.045652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:42.045652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:42.045847 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:42.045935 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:42.046074 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 07:09:42.046148 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 07:09:42.049855 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35085
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:42.131492 1722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:42.131708 1722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:42.131731 1722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:42.132936 1722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:42.132978 1722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:42.132999 1722 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:09:42.133018 1722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:09:42.134682 1722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:42.134931 1722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:42.135155 1722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:42.140089 1730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:42.140108 1727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:42.140094 1728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:42.140362 1722 server_base.cc:1034] running on GCE node
I20260924 07:09:42.140589 1722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:42.140875 1722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:42.142074 1722 hybrid_clock.cc:648] HybridClock initialized: now 1790233782142062 us; error 49 us; skew 500 ppm
Sep 24 07:09:42 dist-test-slave-14ww krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:42.144688 1722 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:09:42.145174 1722 webserver.cc:460] Webserver started at http://127.0.98.2:36593/ using document root <none> and password file <none>
I20260924 07:09:42.145545 1722 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:42.145639 1722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:42.145805 1722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:42.146869 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-1/data/instance:
uuid: "c4f57c71a456480fb2168fabd2513952"
format_stamp: "Formatted at 2026-09-24 07:09:42 on dist-test-slave-14ww"
server_key: "e5082dd32c75c794c950e251865e12a0"
server_key_iv: "1967f76a164afd26b644bc39083e368f"
server_key_version: "encryptionkey@0"
I20260924 07:09:42.147245 1722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c4f57c71a456480fb2168fabd2513952"
format_stamp: "Formatted at 2026-09-24 07:09:42 on dist-test-slave-14ww"
server_key: "e5082dd32c75c794c950e251865e12a0"
server_key_iv: "1967f76a164afd26b644bc39083e368f"
server_key_version: "encryptionkey@0"
I20260924 07:09:42.148638 1722 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 07:09:42.149549 1737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:42.149796 1722 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:09:42.149916 1722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-1/wal
uuid: "c4f57c71a456480fb2168fabd2513952"
format_stamp: "Formatted at 2026-09-24 07:09:42 on dist-test-slave-14ww"
server_key: "e5082dd32c75c794c950e251865e12a0"
server_key_iv: "1967f76a164afd26b644bc39083e368f"
server_key_version: "encryptionkey@0"
I20260924 07:09:42.149995 1722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:42.178057 1722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:42.180078 1722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:42.180235 1722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:42.180791 1722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:42.180862 1722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:42.180903 1722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:42.180931 1722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 07:09:42 dist-test-slave-14ww krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:09:42.188342 1722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36799
I20260924 07:09:42.188373 1852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36799 every 8 connection(s)
I20260924 07:09:42.188786 1722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:09:42.190692 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.181299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50667 (local address 127.0.98.62:35085)
0924 07:09:42.181343 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:42.181345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:42.181544 (+ 199us) server_negotiation.cc:407] Connection header received
0924 07:09:42.182031 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:42.182033 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:42.182049 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:42.182127 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:42.182765 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.183640 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.185059 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.185401 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.185597 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:42.187996 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:42.187997 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:42.187997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:42.188005 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:42.189616 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:42.189966 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:42.189967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:42.189967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:42.190019 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:42.190210 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:42.190210 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:42.190210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:42.190317 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:42.190365 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:42.190576 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 07:09:42.190631 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 07:09:42.190869 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.181291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35085 (local address 127.0.98.2:50667)
0924 07:09:42.181434 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:42.181467 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:42.181927 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:42.182191 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:42.182195 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:42.182452 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:42.182686 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:42.182691 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.183749 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.183750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:42.184989 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:42.184991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.185645 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.185647 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:42.185765 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:42.186363 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:42.186370 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:42.187905 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:42.189690 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:42.189695 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:42.189696 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:42.189900 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:42.190080 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:42.190081 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:42.190081 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:42.190170 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:42.190370 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:42.190374 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:42.190478 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 07:09:42.190563 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 07:09:42.190673 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":77,"threads_started":1}
I20260924 07:09:42.191340 1853 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35085
I20260924 07:09:42.191422 1853 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:42.191593 1853 heartbeater.cc:511] Master 127.0.98.62:35085 requested a full tablet report, sending...
I20260924 07:09:42.192034 1510 ts_manager.cc:194] Registered new tserver with Master: c4f57c71a456480fb2168fabd2513952 (127.0.98.2:36799)
I20260924 07:09:42.192524 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50667
I20260924 07:09:42.194772 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 1722
I20260924 07:09:42.194922 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:09:42.195111 392 external_mini_cluster.cc:1428] Setting key cf2207f9065fedbee37ac87bac74388a
I20260924 07:09:42.198480 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.192461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50571 (local address 127.0.98.62:35085)
0924 07:09:42.192504 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:42.192506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:42.192572 (+ 66us) server_negotiation.cc:407] Connection header received
0924 07:09:42.192641 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:42.192641 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:42.192653 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:42.192703 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:42.193088 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.193634 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.194391 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.194596 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.194677 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:42.195638 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:42.195640 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:42.195640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:42.195647 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:42.197319 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:42.197643 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:42.197644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:42.197645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:42.197710 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:42.197891 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:42.197892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:42.197892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:42.198072 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:42.198147 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:42.198317 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 07:09:42.198391 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 07:09:42.198493 1859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.192395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35085 (local address 127.0.98.2:50571)
0924 07:09:42.192519 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:42.192530 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:42.192586 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:42.192712 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:42.192713 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:42.192950 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:42.193010 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:42.193011 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.193731 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.193732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:42.194295 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:42.194297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.194678 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.194679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:42.194743 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:42.195199 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:42.195204 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:42.195543 (+ 339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:42.197407 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:42.197408 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:42.197408 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:42.197569 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:42.197775 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:42.197775 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:42.197776 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:42.197834 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:42.198152 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:42.198153 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:42.198197 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 07:09:42.198290 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 07:09:42.198390 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":67,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 07:09:42.220176 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35085
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:42.299378 1862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:42.299562 1862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:42.299585 1862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:42.300768 1862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:42.300812 1862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:42.300832 1862 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:09:42.300849 1862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:09:42.302309 1862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:42.302528 1862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:42.302752 1862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:09:42.307838 1862 server_base.cc:1034] running on GCE node
W20260924 07:09:42.307758 1870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:42.307758 1868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:42.307747 1867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:42.308218 1862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:42.308466 1862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:42.309614 1862 hybrid_clock.cc:648] HybridClock initialized: now 1790233782309616 us; error 40 us; skew 500 ppm
Sep 24 07:09:42 dist-test-slave-14ww krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:42.312201 1862 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:09:42.312625 1862 webserver.cc:460] Webserver started at http://127.0.98.3:34705/ using document root <none> and password file <none>
I20260924 07:09:42.312898 1862 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:42.312966 1862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:42.313115 1862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:42.314185 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-2/data/instance:
uuid: "4c1669f3864049dbad5d2004b0a37145"
format_stamp: "Formatted at 2026-09-24 07:09:42 on dist-test-slave-14ww"
server_key: "8537be98baff99f49b42761e99132b26"
server_key_iv: "4ee8a232160fea680b23b7f16cbd880d"
server_key_version: "encryptionkey@0"
I20260924 07:09:42.314545 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4c1669f3864049dbad5d2004b0a37145"
format_stamp: "Formatted at 2026-09-24 07:09:42 on dist-test-slave-14ww"
server_key: "8537be98baff99f49b42761e99132b26"
server_key_iv: "4ee8a232160fea680b23b7f16cbd880d"
server_key_version: "encryptionkey@0"
I20260924 07:09:42.315872 1862 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:09:42.316638 1877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:42.316845 1862 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:09:42.316942 1862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-2/wal
uuid: "4c1669f3864049dbad5d2004b0a37145"
format_stamp: "Formatted at 2026-09-24 07:09:42 on dist-test-slave-14ww"
server_key: "8537be98baff99f49b42761e99132b26"
server_key_iv: "4ee8a232160fea680b23b7f16cbd880d"
server_key_version: "encryptionkey@0"
I20260924 07:09:42.317025 1862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:42.327028 1862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:42.329759 1862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:42.329908 1862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:42.330585 1862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:42.330629 1862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:42.330662 1862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:42.330713 1862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:42.337998 1862 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43181
I20260924 07:09:42.338423 1862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 07:09:42 dist-test-slave-14ww krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:09:42.340139 1992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43181 every 8 connection(s)
I20260924 07:09:42.342690 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.330867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33117 (local address 127.0.98.62:35085)
0924 07:09:42.330914 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:42.330916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:42.331410 (+ 494us) server_negotiation.cc:407] Connection header received
0924 07:09:42.332141 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:42.332143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:42.332155 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:42.332209 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:42.333750 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.334611 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.336311 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.336617 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.336668 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:42.339470 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:42.339472 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:42.339473 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:42.339480 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:42.341226 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:42.341714 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:42.341716 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:42.341716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:42.341780 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:42.342038 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:42.342039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:42.342039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:42.342157 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:42.342233 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:42.342498 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 07:09:42.342576 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 07:09:42.342846 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.331026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35085 (local address 127.0.98.3:33117)
0924 07:09:42.331272 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:42.331323 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:42.332014 (+ 691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:42.332914 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:42.332919 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:42.333314 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:42.333619 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:42.333624 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.334738 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.334739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:42.336194 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:42.336197 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.336788 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.336789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:42.336944 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:42.337631 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:42.337641 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:42.339371 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:42.341327 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:42.341333 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:42.341337 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:42.341639 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:42.341852 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:42.341853 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:42.341853 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:42.341963 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:42.342223 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:42.342229 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:42.342371 (+ 142us) client_negotiation.cc:769] Sending connection context
0924 07:09:42.342478 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 07:09:42.342624 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":78,"threads_started":1}
I20260924 07:09:42.343312 1993 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35085
I20260924 07:09:42.343396 1993 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:42.343570 1993 heartbeater.cc:511] Master 127.0.98.62:35085 requested a full tablet report, sending...
I20260924 07:09:42.343940 1510 ts_manager.cc:194] Registered new tserver with Master: 4c1669f3864049dbad5d2004b0a37145 (127.0.98.3:43181)
I20260924 07:09:42.344457 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33117
I20260924 07:09:42.346891 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 1862
I20260924 07:09:42.346978 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:09:42.347162 392 external_mini_cluster.cc:1428] Setting key af1d94b290d5b3deb1685c34b339010c
I20260924 07:09:42.348030 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:09:42.349766 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.344372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41785 (local address 127.0.98.62:35085)
0924 07:09:42.344415 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:42.344417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:42.344462 (+ 45us) server_negotiation.cc:407] Connection header received
0924 07:09:42.344538 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:42.344538 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:42.344548 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:42.344604 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:42.345001 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.345456 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.346141 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.346355 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.346428 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:42.347302 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:42.347304 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:42.347304 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:42.347311 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:42.348664 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:42.349034 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:42.349035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:42.349036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:42.349084 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:42.349291 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:42.349292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:42.349292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:42.349397 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:42.349470 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:42.349623 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 07:09:42.349686 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 07:09:42.349797 1999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.344290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35085 (local address 127.0.98.3:41785)
0924 07:09:42.344405 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:42.344415 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:42.344479 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:42.344643 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:42.344644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:42.344861 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:42.344912 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:42.344913 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.345545 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.345546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:42.346059 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:42.346060 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.346427 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.346428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:42.346493 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:42.346906 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:42.346910 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:42.347234 (+ 324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:42.348785 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:42.348786 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:42.348786 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:42.348946 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:42.349166 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:42.349167 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:42.349167 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:42.349227 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:42.349463 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:42.349464 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:42.349504 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 07:09:42.349586 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 07:09:42.349681 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":64,"threads_started":1}
Sep 24 07:09:42 dist-test-slave-14ww krb5kdc[1465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233782, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 07:09:42 dist-test-slave-14ww krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233782, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:09:42.369633 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.363214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36164 (local address 127.0.98.62:35085)
0924 07:09:42.363274 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:42.363276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:42.363309 (+ 33us) server_negotiation.cc:407] Connection header received
0924 07:09:42.363337 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:42.363338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:42.363346 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:42.363386 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:42.363883 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.364354 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.365150 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.365430 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.365494 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:42.367519 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:42.367520 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:42.367520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:42.367526 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:42.368711 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:42.369055 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:42.369055 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:42.369056 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:42.369103 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:42.369281 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:42.369281 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:42.369281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:42.369372 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:42.369446 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:42.369528 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 07:09:42.369578 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 07:09:42.371222 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36164:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:09:42.372812 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:09:42.379071 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.375020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36799 (local address 127.0.0.1:60666)
0924 07:09:42.375213 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:42.375252 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:42.375319 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:42.375683 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:42.375684 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:42.375686 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:42.375745 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:42.375747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.377101 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.377101 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:42.378000 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:42.378002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.378773 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.378775 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:42.378853 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:42.378867 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 07:09:42.378936 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 07:09:42.378996 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":64,"threads_started":1}
I20260924 07:09:42.379081 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.375135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60666 (local address 127.0.98.2:36799)
0924 07:09:42.375498 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:42.375503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:42.375512 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:09:42.375555 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:42.375558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:42.375574 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:42.375832 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:42.375891 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.377004 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.378065 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.378663 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.378767 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:42.378847 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:42.378950 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 07:09:42.378991 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":92,"threads_started":1}
I20260924 07:09:42.379357 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.375352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41540 (local address 127.0.98.3:43181)
0924 07:09:42.375492 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:42.375497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:42.375507 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:09:42.375545 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:42.375547 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:42.375563 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:42.375665 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:42.375884 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.376591 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.377995 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.378730 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.378991 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:42.379068 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:42.379093 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 07:09:42.379265 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
I20260924 07:09:42.379354 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.375224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46873 (local address 127.0.0.1:55794)
0924 07:09:42.375512 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:42.375522 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:42.375563 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:42.375773 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:42.375774 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:42.375776 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:42.375829 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:42.375831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.377072 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.377072 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:42.378153 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:42.378155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.379100 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.379101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:42.379173 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:42.379177 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:09:42.379232 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 07:09:42.379271 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":30,"thread_start_us":68,"threads_started":1}
I20260924 07:09:42.379488 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.375076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43181 (local address 127.0.0.1:41540)
0924 07:09:42.375308 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:42.375318 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:42.375371 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:42.375674 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:42.375677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:42.375679 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:42.375805 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:42.375808 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.376679 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.376680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:42.377905 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:42.377908 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.378842 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.378844 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:42.378906 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:42.378910 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:09:42.378942 (+ 32us) client_negotiation.cc:240] Negotiation successful
0924 07:09:42.379426 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":433,"thread_start_us":86,"threads_started":1}
I20260924 07:09:42.379663 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.375383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55794 (local address 127.0.98.1:46873)
0924 07:09:42.375559 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:42.375562 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:42.375572 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:09:42.375633 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:42.375634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:42.375650 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:42.375717 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:42.375903 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.376986 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.378242 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.378937 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.378995 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:42.379067 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:42.379510 (+ 443us) server_negotiation.cc:299] Negotiation successful
0924 07:09:42.379554 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":78,"threads_started":1}
I20260924 07:09:42.379824 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 1dfe43c0958b4e518f3db28d7496e150 (DEFAULT_TABLE table=test-table [id=dd36b7eaa8ae40a1ba6401d03978e1b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:42.380174 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dfe43c0958b4e518f3db28d7496e150. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:42.380129 1921 tablet_service.cc:1467] Processing CreateTablet for tablet 1dfe43c0958b4e518f3db28d7496e150 (DEFAULT_TABLE table=test-table [id=dd36b7eaa8ae40a1ba6401d03978e1b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:42.380362 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dfe43c0958b4e518f3db28d7496e150. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:42.380259 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 1dfe43c0958b4e518f3db28d7496e150 (DEFAULT_TABLE table=test-table [id=dd36b7eaa8ae40a1ba6401d03978e1b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:42.380537 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dfe43c0958b4e518f3db28d7496e150. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:42.383013 2016 tablet_bootstrap.cc:492] T 1dfe43c0958b4e518f3db28d7496e150 P 4c1669f3864049dbad5d2004b0a37145: Bootstrap starting.
I20260924 07:09:42.383527 2016 tablet_bootstrap.cc:654] T 1dfe43c0958b4e518f3db28d7496e150 P 4c1669f3864049dbad5d2004b0a37145: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:42.383812 2016 log.cc:824] T 1dfe43c0958b4e518f3db28d7496e150 P 4c1669f3864049dbad5d2004b0a37145: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:42.383993 2017 tablet_bootstrap.cc:492] T 1dfe43c0958b4e518f3db28d7496e150 P c4f57c71a456480fb2168fabd2513952: Bootstrap starting.
I20260924 07:09:42.384536 2018 tablet_bootstrap.cc:492] T 1dfe43c0958b4e518f3db28d7496e150 P f1ab124407d1401187fcc2295578d2ad: Bootstrap starting.
I20260924 07:09:42.384569 2017 tablet_bootstrap.cc:654] T 1dfe43c0958b4e518f3db28d7496e150 P c4f57c71a456480fb2168fabd2513952: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:42.384891 2016 tablet_bootstrap.cc:492] T 1dfe43c0958b4e518f3db28d7496e150 P 4c1669f3864049dbad5d2004b0a37145: No bootstrap required, opened a new log
I20260924 07:09:42.384925 2017 log.cc:824] T 1dfe43c0958b4e518f3db28d7496e150 P c4f57c71a456480fb2168fabd2513952: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:42.384961 2016 ts_tablet_manager.cc:1397] T 1dfe43c0958b4e518f3db28d7496e150 P 4c1669f3864049dbad5d2004b0a37145: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:09:42.385180 2018 tablet_bootstrap.cc:654] T 1dfe43c0958b4e518f3db28d7496e150 P f1ab124407d1401187fcc2295578d2ad: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:42.385497 2018 log.cc:824] T 1dfe43c0958b4e518f3db28d7496e150 P f1ab124407d1401187fcc2295578d2ad: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:42.385682 2017 tablet_bootstrap.cc:492] T 1dfe43c0958b4e518f3db28d7496e150 P c4f57c71a456480fb2168fabd2513952: No bootstrap required, opened a new log
I20260924 07:09:42.385746 2017 ts_tablet_manager.cc:1397] T 1dfe43c0958b4e518f3db28d7496e150 P c4f57c71a456480fb2168fabd2513952: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 07:09:42.386304 2018 tablet_bootstrap.cc:492] T 1dfe43c0958b4e518f3db28d7496e150 P f1ab124407d1401187fcc2295578d2ad: No bootstrap required, opened a new log
I20260924 07:09:42.386370 2018 ts_tablet_manager.cc:1397] T 1dfe43c0958b4e518f3db28d7496e150 P f1ab124407d1401187fcc2295578d2ad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:09:42.387053 2016 raft_consensus.cc:348] T 1dfe43c0958b4e518f3db28d7496e150 P 4c1669f3864049dbad5d2004b0a37145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 } } peers { permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 } } peers { permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 } }
I20260924 07:09:42.387220 2016 raft_consensus.cc:374] T 1dfe43c0958b4e518f3db28d7496e150 P 4c1669f3864049dbad5d2004b0a37145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:42.387260 2016 raft_consensus.cc:729] T 1dfe43c0958b4e518f3db28d7496e150 P 4c1669f3864049dbad5d2004b0a37145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1669f3864049dbad5d2004b0a37145, State: Initialized, Role: FOLLOWER
I20260924 07:09:42.387394 2016 consensus_queue.cc:260] T 1dfe43c0958b4e518f3db28d7496e150 P 4c1669f3864049dbad5d2004b0a37145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 } } peers { permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 } } peers { permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 } }
I20260924 07:09:42.387646 2016 ts_tablet_manager.cc:1428] T 1dfe43c0958b4e518f3db28d7496e150 P 4c1669f3864049dbad5d2004b0a37145: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 07:09:42.387777 1993 heartbeater.cc:503] Master 127.0.98.62:35085 was elected leader, sending a full tablet report...
I20260924 07:09:42.387738 2017 raft_consensus.cc:348] T 1dfe43c0958b4e518f3db28d7496e150 P c4f57c71a456480fb2168fabd2513952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 } } peers { permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 } } peers { permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 } }
I20260924 07:09:42.387876 2017 raft_consensus.cc:374] T 1dfe43c0958b4e518f3db28d7496e150 P c4f57c71a456480fb2168fabd2513952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:42.387898 2017 raft_consensus.cc:729] T 1dfe43c0958b4e518f3db28d7496e150 P c4f57c71a456480fb2168fabd2513952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f57c71a456480fb2168fabd2513952, State: Initialized, Role: FOLLOWER
I20260924 07:09:42.388002 2017 consensus_queue.cc:260] T 1dfe43c0958b4e518f3db28d7496e150 P c4f57c71a456480fb2168fabd2513952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 } } peers { permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 } } peers { permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 } }
I20260924 07:09:42.388242 2017 ts_tablet_manager.cc:1428] T 1dfe43c0958b4e518f3db28d7496e150 P c4f57c71a456480fb2168fabd2513952: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 07:09:42.388257 1853 heartbeater.cc:503] Master 127.0.98.62:35085 was elected leader, sending a full tablet report...
I20260924 07:09:42.388221 2018 raft_consensus.cc:348] T 1dfe43c0958b4e518f3db28d7496e150 P f1ab124407d1401187fcc2295578d2ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 } } peers { permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 } } peers { permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 } }
I20260924 07:09:42.388361 2018 raft_consensus.cc:374] T 1dfe43c0958b4e518f3db28d7496e150 P f1ab124407d1401187fcc2295578d2ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:42.388403 2018 raft_consensus.cc:729] T 1dfe43c0958b4e518f3db28d7496e150 P f1ab124407d1401187fcc2295578d2ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1ab124407d1401187fcc2295578d2ad, State: Initialized, Role: FOLLOWER
I20260924 07:09:42.388516 2018 consensus_queue.cc:260] T 1dfe43c0958b4e518f3db28d7496e150 P f1ab124407d1401187fcc2295578d2ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 } } peers { permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 } } peers { permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 } }
I20260924 07:09:42.388780 2018 ts_tablet_manager.cc:1428] T 1dfe43c0958b4e518f3db28d7496e150 P f1ab124407d1401187fcc2295578d2ad: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 07:09:42.388901 1713 heartbeater.cc:503] Master 127.0.98.62:35085 was elected leader, sending a full tablet report...
I20260924 07:09:42.409269 2023 raft_consensus.cc:482] T 1dfe43c0958b4e518f3db28d7496e150 P c4f57c71a456480fb2168fabd2513952 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:09:42.409375 2023 raft_consensus.cc:504] T 1dfe43c0958b4e518f3db28d7496e150 P c4f57c71a456480fb2168fabd2513952 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 } } peers { permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 } } peers { permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 } }
I20260924 07:09:42.409709 2023 leader_election.cc:288] T 1dfe43c0958b4e518f3db28d7496e150 P c4f57c71a456480fb2168fabd2513952 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1ab124407d1401187fcc2295578d2ad (127.0.98.1:46873), 4c1669f3864049dbad5d2004b0a37145 (127.0.98.3:43181)
I20260924 07:09:42.413756 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.409997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38109 (local address 127.0.98.1:46873)
0924 07:09:42.410046 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:42.410049 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:42.410214 (+ 165us) server_negotiation.cc:407] Connection header received
0924 07:09:42.410239 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:42.410240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:42.410250 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:42.410303 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:42.410551 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.411271 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.412819 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.413351 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.413426 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:42.413434 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:42.413659 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 07:09:42.413694 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 07:09:42.413756 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.409991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58403 (local address 127.0.98.3:43181)
0924 07:09:42.410042 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:42.410044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:42.410054 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:09:42.410076 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:42.410077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:42.410085 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:42.410200 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:42.410549 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.411269 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.412819 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.413352 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.413426 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:42.413434 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:42.413658 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 07:09:42.413694 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 07:09:42.413795 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.409907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43181 (local address 127.0.98.2:58403)
0924 07:09:42.409955 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:42.409966 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:42.410008 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:42.410235 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:42.410235 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:42.410237 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:42.410337 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:42.410339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.411368 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.411369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:42.412695 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:42.412698 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.413463 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.413464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:42.413564 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:42.413571 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 07:09:42.413644 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 07:09:42.413697 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":84}
I20260924 07:09:42.413838 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.409907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46873 (local address 127.0.98.2:38109)
0924 07:09:42.410093 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:42.410101 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:42.410151 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:42.410325 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:42.410326 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:42.410327 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:42.410380 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:42.410381 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.411368 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.411369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:42.412696 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:42.412698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.413463 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.413464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:42.413564 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:42.413571 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 07:09:42.413654 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 07:09:42.413697 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":133,"thread_start_us":80,"threads_started":1}
I20260924 07:09:42.414117 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dfe43c0958b4e518f3db28d7496e150" candidate_uuid: "c4f57c71a456480fb2168fabd2513952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1ab124407d1401187fcc2295578d2ad" is_pre_election: true
I20260924 07:09:42.414141 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dfe43c0958b4e518f3db28d7496e150" candidate_uuid: "c4f57c71a456480fb2168fabd2513952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1669f3864049dbad5d2004b0a37145" is_pre_election: true
I20260924 07:09:42.414242 1667 raft_consensus.cc:2454] T 1dfe43c0958b4e518f3db28d7496e150 P f1ab124407d1401187fcc2295578d2ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4f57c71a456480fb2168fabd2513952 in term 0.
I20260924 07:09:42.414287 1947 raft_consensus.cc:2454] T 1dfe43c0958b4e518f3db28d7496e150 P 4c1669f3864049dbad5d2004b0a37145 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4f57c71a456480fb2168fabd2513952 in term 0.
I20260924 07:09:42.414476 1741 leader_election.cc:302] T 1dfe43c0958b4e518f3db28d7496e150 P c4f57c71a456480fb2168fabd2513952 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f57c71a456480fb2168fabd2513952, f1ab124407d1401187fcc2295578d2ad; no voters:
I20260924 07:09:42.414633 2023 raft_consensus.cc:2781] T 1dfe43c0958b4e518f3db28d7496e150 P c4f57c71a456480fb2168fabd2513952 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:09:42.414714 2023 raft_consensus.cc:482] T 1dfe43c0958b4e518f3db28d7496e150 P c4f57c71a456480fb2168fabd2513952 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:09:42.414747 2023 raft_consensus.cc:3037] T 1dfe43c0958b4e518f3db28d7496e150 P c4f57c71a456480fb2168fabd2513952 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:42.415628 2023 raft_consensus.cc:504] T 1dfe43c0958b4e518f3db28d7496e150 P c4f57c71a456480fb2168fabd2513952 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 } } peers { permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 } } peers { permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 } }
I20260924 07:09:42.415771 2023 leader_election.cc:288] T 1dfe43c0958b4e518f3db28d7496e150 P c4f57c71a456480fb2168fabd2513952 [CANDIDATE]: Term 1 election: Requested vote from peers f1ab124407d1401187fcc2295578d2ad (127.0.98.1:46873), 4c1669f3864049dbad5d2004b0a37145 (127.0.98.3:43181)
I20260924 07:09:42.415889 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dfe43c0958b4e518f3db28d7496e150" candidate_uuid: "c4f57c71a456480fb2168fabd2513952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1ab124407d1401187fcc2295578d2ad"
I20260924 07:09:42.415966 1667 raft_consensus.cc:3037] T 1dfe43c0958b4e518f3db28d7496e150 P f1ab124407d1401187fcc2295578d2ad [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:42.415982 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dfe43c0958b4e518f3db28d7496e150" candidate_uuid: "c4f57c71a456480fb2168fabd2513952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1669f3864049dbad5d2004b0a37145"
I20260924 07:09:42.416034 1947 raft_consensus.cc:3037] T 1dfe43c0958b4e518f3db28d7496e150 P 4c1669f3864049dbad5d2004b0a37145 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:42.416795 1667 raft_consensus.cc:2454] T 1dfe43c0958b4e518f3db28d7496e150 P f1ab124407d1401187fcc2295578d2ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4f57c71a456480fb2168fabd2513952 in term 1.
I20260924 07:09:42.416795 1947 raft_consensus.cc:2454] T 1dfe43c0958b4e518f3db28d7496e150 P 4c1669f3864049dbad5d2004b0a37145 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4f57c71a456480fb2168fabd2513952 in term 1.
I20260924 07:09:42.416957 1738 leader_election.cc:302] T 1dfe43c0958b4e518f3db28d7496e150 P c4f57c71a456480fb2168fabd2513952 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c1669f3864049dbad5d2004b0a37145, c4f57c71a456480fb2168fabd2513952; no voters:
I20260924 07:09:42.417085 2023 raft_consensus.cc:2781] T 1dfe43c0958b4e518f3db28d7496e150 P c4f57c71a456480fb2168fabd2513952 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:09:42.417245 2023 raft_consensus.cc:686] T 1dfe43c0958b4e518f3db28d7496e150 P c4f57c71a456480fb2168fabd2513952 [term 1 LEADER]: Becoming Leader. State: Replica: c4f57c71a456480fb2168fabd2513952, State: Running, Role: LEADER
I20260924 07:09:42.417361 2023 consensus_queue.cc:237] T 1dfe43c0958b4e518f3db28d7496e150 P c4f57c71a456480fb2168fabd2513952 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 } } peers { permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 } } peers { permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 } }
I20260924 07:09:42.418094 1509 catalog_manager.cc:5495] T 1dfe43c0958b4e518f3db28d7496e150 P c4f57c71a456480fb2168fabd2513952 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4f57c71a456480fb2168fabd2513952 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c4f57c71a456480fb2168fabd2513952" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 } health_report { overall_health: UNKNOWN } } }
I20260924 07:09:42.430266 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.427747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60670 (local address 127.0.98.2:36799)
0924 07:09:42.427793 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:42.427795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:42.427850 (+ 55us) server_negotiation.cc:407] Connection header received
0924 07:09:42.427938 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:42.427939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:42.427960 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:42.428016 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:09:42.428109 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.428572 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.429506 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.429694 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.429748 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:42.429930 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:09:42.430018 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:09:42.430079 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:42.430135 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 07:09:42.430156 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 07:09:42.432026 1947 raft_consensus.cc:1261] T 1dfe43c0958b4e518f3db28d7496e150 P 4c1669f3864049dbad5d2004b0a37145 [term 1 FOLLOWER]: Refusing update from remote peer c4f57c71a456480fb2168fabd2513952: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:09:42.432032 1667 raft_consensus.cc:1261] T 1dfe43c0958b4e518f3db28d7496e150 P f1ab124407d1401187fcc2295578d2ad [term 1 FOLLOWER]: Refusing update from remote peer c4f57c71a456480fb2168fabd2513952: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:09:42.432333 2023 consensus_queue.cc:1035] T 1dfe43c0958b4e518f3db28d7496e150 P c4f57c71a456480fb2168fabd2513952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:09:42.432499 2026 consensus_queue.cc:1035] T 1dfe43c0958b4e518f3db28d7496e150 P c4f57c71a456480fb2168fabd2513952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:09:42.435254 2029 mvcc.cc:205] Tried to move back new op lower bound from 7332797572839006208 to 7332797572782051328. Current Snapshot: MvccSnapshot[applied={T|T < 7332797572839006208 or (T in {7332797572839006208})}]
I20260924 07:09:42.436625 2030 mvcc.cc:205] Tried to move back new op lower bound from 7332797572839006208 to 7332797572782051328. Current Snapshot: MvccSnapshot[applied={T|T < 7332797572839006208 or (T in {7332797572839006208})}]
I20260924 07:09:42.438191 1509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:09:42.438289 1509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:09:42.439008 1509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5528fa3ab4b145ebbc6cd8f9eb434d15: Sending DeleteTablet for 3 replicas of tablet 1dfe43c0958b4e518f3db28d7496e150
I20260924 07:09:42.439296 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dfe43c0958b4e518f3db28d7496e150 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60666
I20260924 07:09:42.439324 1921 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dfe43c0958b4e518f3db28d7496e150 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41540
I20260924 07:09:42.439534 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dfe43c0958b4e518f3db28d7496e150 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55794
I20260924 07:09:42.439744 2039 tablet_replica.cc:330] T 1dfe43c0958b4e518f3db28d7496e150 P 4c1669f3864049dbad5d2004b0a37145: stopping tablet replica
I20260924 07:09:42.439741 2040 tablet_replica.cc:330] T 1dfe43c0958b4e518f3db28d7496e150 P f1ab124407d1401187fcc2295578d2ad: stopping tablet replica
I20260924 07:09:42.439762 1509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36164:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 07:09:42.439844 2039 raft_consensus.cc:2229] T 1dfe43c0958b4e518f3db28d7496e150 P 4c1669f3864049dbad5d2004b0a37145 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260924 07:09:42.439903 1509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:09:42.439942 2039 raft_consensus.cc:2258] T 1dfe43c0958b4e518f3db28d7496e150 P 4c1669f3864049dbad5d2004b0a37145 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:09:42.440121 2040 raft_consensus.cc:2229] T 1dfe43c0958b4e518f3db28d7496e150 P f1ab124407d1401187fcc2295578d2ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:09:42.440244 2040 raft_consensus.cc:2258] T 1dfe43c0958b4e518f3db28d7496e150 P f1ab124407d1401187fcc2295578d2ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:09:42.440778 2039 ts_tablet_manager.cc:1905] T 1dfe43c0958b4e518f3db28d7496e150 P 4c1669f3864049dbad5d2004b0a37145: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:09:42.441139 2040 ts_tablet_manager.cc:1905] T 1dfe43c0958b4e518f3db28d7496e150 P f1ab124407d1401187fcc2295578d2ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:09:42.441391 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 574ae28683784d6c8db0bf88de2d822f (DEFAULT_TABLE table=test-table [id=575a4a05926a4f178768056259e3e386]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:42.441421 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 574ae28683784d6c8db0bf88de2d822f (DEFAULT_TABLE table=test-table [id=575a4a05926a4f178768056259e3e386]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:42.441495 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 574ae28683784d6c8db0bf88de2d822f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:42.441524 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 574ae28683784d6c8db0bf88de2d822f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:42.442863 2017 tablet_bootstrap.cc:492] T 574ae28683784d6c8db0bf88de2d822f P c4f57c71a456480fb2168fabd2513952: Bootstrap starting.
I20260924 07:09:42.442919 2038 tablet_replica.cc:330] T 1dfe43c0958b4e518f3db28d7496e150 P c4f57c71a456480fb2168fabd2513952: stopping tablet replica
I20260924 07:09:42.443006 2038 raft_consensus.cc:2229] T 1dfe43c0958b4e518f3db28d7496e150 P c4f57c71a456480fb2168fabd2513952 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:09:42.443135 2038 raft_consensus.cc:2258] T 1dfe43c0958b4e518f3db28d7496e150 P c4f57c71a456480fb2168fabd2513952 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:09:42.443287 2017 tablet_bootstrap.cc:654] T 574ae28683784d6c8db0bf88de2d822f P c4f57c71a456480fb2168fabd2513952: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:42.443557 1921 tablet_service.cc:1467] Processing CreateTablet for tablet 574ae28683784d6c8db0bf88de2d822f (DEFAULT_TABLE table=test-table [id=575a4a05926a4f178768056259e3e386]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:42.443624 2018 tablet_bootstrap.cc:492] T 574ae28683784d6c8db0bf88de2d822f P f1ab124407d1401187fcc2295578d2ad: Bootstrap starting.
I20260924 07:09:42.444015 2018 tablet_bootstrap.cc:654] T 574ae28683784d6c8db0bf88de2d822f P f1ab124407d1401187fcc2295578d2ad: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:42.444689 2018 tablet_bootstrap.cc:492] T 574ae28683784d6c8db0bf88de2d822f P f1ab124407d1401187fcc2295578d2ad: No bootstrap required, opened a new log
I20260924 07:09:42.444772 2018 ts_tablet_manager.cc:1397] T 574ae28683784d6c8db0bf88de2d822f P f1ab124407d1401187fcc2295578d2ad: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:09:42.444940 2018 raft_consensus.cc:348] T 574ae28683784d6c8db0bf88de2d822f P f1ab124407d1401187fcc2295578d2ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 } } peers { permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 } } peers { permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 } }
I20260924 07:09:42.444960 2038 ts_tablet_manager.cc:1905] T 1dfe43c0958b4e518f3db28d7496e150 P c4f57c71a456480fb2168fabd2513952: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:09:42.445026 2018 raft_consensus.cc:374] T 574ae28683784d6c8db0bf88de2d822f P f1ab124407d1401187fcc2295578d2ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:42.445055 2018 raft_consensus.cc:729] T 574ae28683784d6c8db0bf88de2d822f P f1ab124407d1401187fcc2295578d2ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1ab124407d1401187fcc2295578d2ad, State: Initialized, Role: FOLLOWER
I20260924 07:09:42.445099 2018 consensus_queue.cc:260] T 574ae28683784d6c8db0bf88de2d822f P f1ab124407d1401187fcc2295578d2ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 } } peers { permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 } } peers { permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 } }
I20260924 07:09:42.445174 2018 ts_tablet_manager.cc:1428] T 574ae28683784d6c8db0bf88de2d822f P f1ab124407d1401187fcc2295578d2ad: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:42.445865 2040 ts_tablet_manager.cc:1918] T 1dfe43c0958b4e518f3db28d7496e150 P f1ab124407d1401187fcc2295578d2ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:09:42.445941 2040 log.cc:1196] T 1dfe43c0958b4e518f3db28d7496e150 P f1ab124407d1401187fcc2295578d2ad: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-0/wal/wals/1dfe43c0958b4e518f3db28d7496e150
I20260924 07:09:42.446203 2040 ts_tablet_manager.cc:1939] T 1dfe43c0958b4e518f3db28d7496e150 P f1ab124407d1401187fcc2295578d2ad: Deleting consensus metadata
I20260924 07:09:42.446395 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 574ae28683784d6c8db0bf88de2d822f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:42.446396 2039 ts_tablet_manager.cc:1918] T 1dfe43c0958b4e518f3db28d7496e150 P 4c1669f3864049dbad5d2004b0a37145: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:09:42.446491 2039 log.cc:1196] T 1dfe43c0958b4e518f3db28d7496e150 P 4c1669f3864049dbad5d2004b0a37145: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-2/wal/wals/1dfe43c0958b4e518f3db28d7496e150
I20260924 07:09:42.446761 2039 ts_tablet_manager.cc:1939] T 1dfe43c0958b4e518f3db28d7496e150 P 4c1669f3864049dbad5d2004b0a37145: Deleting consensus metadata
I20260924 07:09:42.446822 1497 catalog_manager.cc:4841] TS f1ab124407d1401187fcc2295578d2ad (127.0.98.1:46873): tablet 1dfe43c0958b4e518f3db28d7496e150 (table test-table [id=dd36b7eaa8ae40a1ba6401d03978e1b7]) successfully deleted
I20260924 07:09:42.447175 1494 catalog_manager.cc:4841] TS 4c1669f3864049dbad5d2004b0a37145 (127.0.98.3:43181): tablet 1dfe43c0958b4e518f3db28d7496e150 (table test-table [id=dd36b7eaa8ae40a1ba6401d03978e1b7]) successfully deleted
I20260924 07:09:42.447629 2016 tablet_bootstrap.cc:492] T 574ae28683784d6c8db0bf88de2d822f P 4c1669f3864049dbad5d2004b0a37145: Bootstrap starting.
I20260924 07:09:42.448033 2016 tablet_bootstrap.cc:654] T 574ae28683784d6c8db0bf88de2d822f P 4c1669f3864049dbad5d2004b0a37145: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:42.449139 2016 tablet_bootstrap.cc:492] T 574ae28683784d6c8db0bf88de2d822f P 4c1669f3864049dbad5d2004b0a37145: No bootstrap required, opened a new log
I20260924 07:09:42.449193 2016 ts_tablet_manager.cc:1397] T 574ae28683784d6c8db0bf88de2d822f P 4c1669f3864049dbad5d2004b0a37145: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:09:42.449348 2016 raft_consensus.cc:348] T 574ae28683784d6c8db0bf88de2d822f P 4c1669f3864049dbad5d2004b0a37145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 } } peers { permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 } } peers { permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 } }
I20260924 07:09:42.449425 2016 raft_consensus.cc:374] T 574ae28683784d6c8db0bf88de2d822f P 4c1669f3864049dbad5d2004b0a37145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:42.449450 2016 raft_consensus.cc:729] T 574ae28683784d6c8db0bf88de2d822f P 4c1669f3864049dbad5d2004b0a37145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1669f3864049dbad5d2004b0a37145, State: Initialized, Role: FOLLOWER
I20260924 07:09:42.449501 2016 consensus_queue.cc:260] T 574ae28683784d6c8db0bf88de2d822f P 4c1669f3864049dbad5d2004b0a37145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 } } peers { permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 } } peers { permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 } }
I20260924 07:09:42.449592 2016 ts_tablet_manager.cc:1428] T 574ae28683784d6c8db0bf88de2d822f P 4c1669f3864049dbad5d2004b0a37145: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 07:09:42.450435 2038 ts_tablet_manager.cc:1918] T 1dfe43c0958b4e518f3db28d7496e150 P c4f57c71a456480fb2168fabd2513952: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:09:42.450518 2038 log.cc:1196] T 1dfe43c0958b4e518f3db28d7496e150 P c4f57c71a456480fb2168fabd2513952: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-1/wal/wals/1dfe43c0958b4e518f3db28d7496e150
I20260924 07:09:42.450745 2038 ts_tablet_manager.cc:1939] T 1dfe43c0958b4e518f3db28d7496e150 P c4f57c71a456480fb2168fabd2513952: Deleting consensus metadata
I20260924 07:09:42.451129 1497 catalog_manager.cc:4841] TS c4f57c71a456480fb2168fabd2513952 (127.0.98.2:36799): tablet 1dfe43c0958b4e518f3db28d7496e150 (table test-table [id=dd36b7eaa8ae40a1ba6401d03978e1b7]) successfully deleted
I20260924 07:09:42.451370 2017 tablet_bootstrap.cc:492] T 574ae28683784d6c8db0bf88de2d822f P c4f57c71a456480fb2168fabd2513952: No bootstrap required, opened a new log
I20260924 07:09:42.451419 2017 ts_tablet_manager.cc:1397] T 574ae28683784d6c8db0bf88de2d822f P c4f57c71a456480fb2168fabd2513952: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s
I20260924 07:09:42.451560 2017 raft_consensus.cc:348] T 574ae28683784d6c8db0bf88de2d822f P c4f57c71a456480fb2168fabd2513952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 } } peers { permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 } } peers { permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 } }
I20260924 07:09:42.451627 2017 raft_consensus.cc:374] T 574ae28683784d6c8db0bf88de2d822f P c4f57c71a456480fb2168fabd2513952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:42.451647 2017 raft_consensus.cc:729] T 574ae28683784d6c8db0bf88de2d822f P c4f57c71a456480fb2168fabd2513952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f57c71a456480fb2168fabd2513952, State: Initialized, Role: FOLLOWER
I20260924 07:09:42.451685 2017 consensus_queue.cc:260] T 574ae28683784d6c8db0bf88de2d822f P c4f57c71a456480fb2168fabd2513952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 } } peers { permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 } } peers { permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 } }
I20260924 07:09:42.451772 2017 ts_tablet_manager.cc:1428] T 574ae28683784d6c8db0bf88de2d822f P c4f57c71a456480fb2168fabd2513952: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:42.473224 2026 raft_consensus.cc:482] T 574ae28683784d6c8db0bf88de2d822f P c4f57c71a456480fb2168fabd2513952 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:09:42.473296 2026 raft_consensus.cc:504] T 574ae28683784d6c8db0bf88de2d822f P c4f57c71a456480fb2168fabd2513952 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 } } peers { permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 } } peers { permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 } }
I20260924 07:09:42.473454 2026 leader_election.cc:288] T 574ae28683784d6c8db0bf88de2d822f P c4f57c71a456480fb2168fabd2513952 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1ab124407d1401187fcc2295578d2ad (127.0.98.1:46873), 4c1669f3864049dbad5d2004b0a37145 (127.0.98.3:43181)
I20260924 07:09:42.473672 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "574ae28683784d6c8db0bf88de2d822f" candidate_uuid: "c4f57c71a456480fb2168fabd2513952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1ab124407d1401187fcc2295578d2ad" is_pre_election: true
I20260924 07:09:42.473663 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "574ae28683784d6c8db0bf88de2d822f" candidate_uuid: "c4f57c71a456480fb2168fabd2513952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1669f3864049dbad5d2004b0a37145" is_pre_election: true
I20260924 07:09:42.473773 1667 raft_consensus.cc:2454] T 574ae28683784d6c8db0bf88de2d822f P f1ab124407d1401187fcc2295578d2ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4f57c71a456480fb2168fabd2513952 in term 0.
I20260924 07:09:42.473773 1947 raft_consensus.cc:2454] T 574ae28683784d6c8db0bf88de2d822f P 4c1669f3864049dbad5d2004b0a37145 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4f57c71a456480fb2168fabd2513952 in term 0.
I20260924 07:09:42.473919 1741 leader_election.cc:302] T 574ae28683784d6c8db0bf88de2d822f P c4f57c71a456480fb2168fabd2513952 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f57c71a456480fb2168fabd2513952, f1ab124407d1401187fcc2295578d2ad; no voters:
I20260924 07:09:42.474004 2026 raft_consensus.cc:2781] T 574ae28683784d6c8db0bf88de2d822f P c4f57c71a456480fb2168fabd2513952 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:09:42.474043 2026 raft_consensus.cc:482] T 574ae28683784d6c8db0bf88de2d822f P c4f57c71a456480fb2168fabd2513952 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:09:42.474063 2026 raft_consensus.cc:3037] T 574ae28683784d6c8db0bf88de2d822f P c4f57c71a456480fb2168fabd2513952 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:42.474736 2026 raft_consensus.cc:504] T 574ae28683784d6c8db0bf88de2d822f P c4f57c71a456480fb2168fabd2513952 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 } } peers { permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 } } peers { permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 } }
I20260924 07:09:42.474862 2026 leader_election.cc:288] T 574ae28683784d6c8db0bf88de2d822f P c4f57c71a456480fb2168fabd2513952 [CANDIDATE]: Term 1 election: Requested vote from peers f1ab124407d1401187fcc2295578d2ad (127.0.98.1:46873), 4c1669f3864049dbad5d2004b0a37145 (127.0.98.3:43181)
I20260924 07:09:42.474994 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "574ae28683784d6c8db0bf88de2d822f" candidate_uuid: "c4f57c71a456480fb2168fabd2513952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1ab124407d1401187fcc2295578d2ad"
I20260924 07:09:42.475018 1947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "574ae28683784d6c8db0bf88de2d822f" candidate_uuid: "c4f57c71a456480fb2168fabd2513952" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c1669f3864049dbad5d2004b0a37145"
I20260924 07:09:42.475085 1947 raft_consensus.cc:3037] T 574ae28683784d6c8db0bf88de2d822f P 4c1669f3864049dbad5d2004b0a37145 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:42.475085 1667 raft_consensus.cc:3037] T 574ae28683784d6c8db0bf88de2d822f P f1ab124407d1401187fcc2295578d2ad [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:42.475654 1667 raft_consensus.cc:2454] T 574ae28683784d6c8db0bf88de2d822f P f1ab124407d1401187fcc2295578d2ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4f57c71a456480fb2168fabd2513952 in term 1.
I20260924 07:09:42.475811 1741 leader_election.cc:302] T 574ae28683784d6c8db0bf88de2d822f P c4f57c71a456480fb2168fabd2513952 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4f57c71a456480fb2168fabd2513952, f1ab124407d1401187fcc2295578d2ad; no voters:
I20260924 07:09:42.475857 1947 raft_consensus.cc:2454] T 574ae28683784d6c8db0bf88de2d822f P 4c1669f3864049dbad5d2004b0a37145 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4f57c71a456480fb2168fabd2513952 in term 1.
I20260924 07:09:42.475915 2026 raft_consensus.cc:2781] T 574ae28683784d6c8db0bf88de2d822f P c4f57c71a456480fb2168fabd2513952 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:09:42.475963 2026 raft_consensus.cc:686] T 574ae28683784d6c8db0bf88de2d822f P c4f57c71a456480fb2168fabd2513952 [term 1 LEADER]: Becoming Leader. State: Replica: c4f57c71a456480fb2168fabd2513952, State: Running, Role: LEADER
I20260924 07:09:42.476014 2026 consensus_queue.cc:237] T 574ae28683784d6c8db0bf88de2d822f P c4f57c71a456480fb2168fabd2513952 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 } } peers { permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 } } peers { permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 } }
I20260924 07:09:42.476651 1510 catalog_manager.cc:5495] T 574ae28683784d6c8db0bf88de2d822f P c4f57c71a456480fb2168fabd2513952 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4f57c71a456480fb2168fabd2513952 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c4f57c71a456480fb2168fabd2513952" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 } health_report { overall_health: UNKNOWN } } }
Sep 24 07:09:42 dist-test-slave-14ww krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233781, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
W20260924 07:09:42.520568 1714 tablet.cc:2369] T 574ae28683784d6c8db0bf88de2d822f P f1ab124407d1401187fcc2295578d2ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:09:42.530092 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.501432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35085 (local address 127.0.0.1:36176)
0924 07:09:42.501607 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:42.501617 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:42.501681 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:42.501832 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:42.501832 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:42.502115 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:42.502158 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:42.502160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.503080 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.503080 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:42.503602 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:42.503603 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.504100 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.504101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:42.504167 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:42.513210 (+ 9043us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:42.513215 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:42.515001 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:42.528990 (+ 13989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:42.528998 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:42.529000 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:42.529220 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:42.529452 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:42.529453 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:42.529453 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:42.529510 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:42.529804 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:42.529809 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:42.529853 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 07:09:42.529906 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 07:09:42.530008 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":92,"threads_started":1}
I20260924 07:09:42.530198 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.501485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36176 (local address 127.0.98.62:35085)
0924 07:09:42.501553 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:42.501555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:42.501693 (+ 138us) server_negotiation.cc:407] Connection header received
0924 07:09:42.501727 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:42.501728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:42.501740 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:42.501812 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:42.502238 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.502991 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.503685 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.503990 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.504079 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:42.515113 (+ 11034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:42.515115 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:42.515115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:42.515123 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:42.528897 (+ 13774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:42.529307 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:42.529308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:42.529309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:42.529378 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:42.529576 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:42.529577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:42.529578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:42.529732 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:42.529807 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:42.529958 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 07:09:42.530076 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":42}
I20260924 07:09:42.533449 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36176:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 07:09:42.533667 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:09:42.535185 1647 tablet_service.cc:1467] Processing CreateTablet for tablet 5541f160d628498a8b8ec9fdf572b1b3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ace2ea5ec1194a1685c6de733e78f4ea]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:09:42.535275 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 5541f160d628498a8b8ec9fdf572b1b3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ace2ea5ec1194a1685c6de733e78f4ea]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:09:42.535301 1921 tablet_service.cc:1467] Processing CreateTablet for tablet 5541f160d628498a8b8ec9fdf572b1b3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ace2ea5ec1194a1685c6de733e78f4ea]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:09:42.535357 1647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5541f160d628498a8b8ec9fdf572b1b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:42.535422 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5541f160d628498a8b8ec9fdf572b1b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:42.535425 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5541f160d628498a8b8ec9fdf572b1b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:42.536916 2016 tablet_bootstrap.cc:492] T 5541f160d628498a8b8ec9fdf572b1b3 P 4c1669f3864049dbad5d2004b0a37145: Bootstrap starting.
I20260924 07:09:42.537071 2017 tablet_bootstrap.cc:492] T 5541f160d628498a8b8ec9fdf572b1b3 P c4f57c71a456480fb2168fabd2513952: Bootstrap starting.
I20260924 07:09:42.537371 2016 tablet_bootstrap.cc:654] T 5541f160d628498a8b8ec9fdf572b1b3 P 4c1669f3864049dbad5d2004b0a37145: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:42.537875 2017 tablet_bootstrap.cc:654] T 5541f160d628498a8b8ec9fdf572b1b3 P c4f57c71a456480fb2168fabd2513952: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:42.537881 2018 tablet_bootstrap.cc:492] T 5541f160d628498a8b8ec9fdf572b1b3 P f1ab124407d1401187fcc2295578d2ad: Bootstrap starting.
I20260924 07:09:42.538277 2018 tablet_bootstrap.cc:654] T 5541f160d628498a8b8ec9fdf572b1b3 P f1ab124407d1401187fcc2295578d2ad: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:42.538686 2017 tablet_bootstrap.cc:492] T 5541f160d628498a8b8ec9fdf572b1b3 P c4f57c71a456480fb2168fabd2513952: No bootstrap required, opened a new log
I20260924 07:09:42.538741 2017 ts_tablet_manager.cc:1397] T 5541f160d628498a8b8ec9fdf572b1b3 P c4f57c71a456480fb2168fabd2513952: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 07:09:42.538823 2016 tablet_bootstrap.cc:492] T 5541f160d628498a8b8ec9fdf572b1b3 P 4c1669f3864049dbad5d2004b0a37145: No bootstrap required, opened a new log
I20260924 07:09:42.538874 2016 ts_tablet_manager.cc:1397] T 5541f160d628498a8b8ec9fdf572b1b3 P 4c1669f3864049dbad5d2004b0a37145: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:09:42.538888 2017 raft_consensus.cc:348] T 5541f160d628498a8b8ec9fdf572b1b3 P c4f57c71a456480fb2168fabd2513952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 } } peers { permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 } } peers { permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 } }
I20260924 07:09:42.538959 2017 raft_consensus.cc:374] T 5541f160d628498a8b8ec9fdf572b1b3 P c4f57c71a456480fb2168fabd2513952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:42.538986 2017 raft_consensus.cc:729] T 5541f160d628498a8b8ec9fdf572b1b3 P c4f57c71a456480fb2168fabd2513952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4f57c71a456480fb2168fabd2513952, State: Initialized, Role: FOLLOWER
I20260924 07:09:42.539032 2017 consensus_queue.cc:260] T 5541f160d628498a8b8ec9fdf572b1b3 P c4f57c71a456480fb2168fabd2513952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 } } peers { permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 } } peers { permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 } }
I20260924 07:09:42.539021 2016 raft_consensus.cc:348] T 5541f160d628498a8b8ec9fdf572b1b3 P 4c1669f3864049dbad5d2004b0a37145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 } } peers { permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 } } peers { permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 } }
I20260924 07:09:42.539088 2016 raft_consensus.cc:374] T 5541f160d628498a8b8ec9fdf572b1b3 P 4c1669f3864049dbad5d2004b0a37145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:42.539115 2016 raft_consensus.cc:729] T 5541f160d628498a8b8ec9fdf572b1b3 P 4c1669f3864049dbad5d2004b0a37145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1669f3864049dbad5d2004b0a37145, State: Initialized, Role: FOLLOWER
I20260924 07:09:42.539125 2017 ts_tablet_manager.cc:1428] T 5541f160d628498a8b8ec9fdf572b1b3 P c4f57c71a456480fb2168fabd2513952: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 07:09:42.539157 2016 consensus_queue.cc:260] T 5541f160d628498a8b8ec9fdf572b1b3 P 4c1669f3864049dbad5d2004b0a37145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 } } peers { permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 } } peers { permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 } }
I20260924 07:09:42.539240 2016 ts_tablet_manager.cc:1428] T 5541f160d628498a8b8ec9fdf572b1b3 P 4c1669f3864049dbad5d2004b0a37145: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 07:09:42.539235 2023 tablet_replica.cc:439] T 5541f160d628498a8b8ec9fdf572b1b3 P c4f57c71a456480fb2168fabd2513952: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 } } peers { permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 } } peers { permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 } } }
I20260924 07:09:42.539299 2023 tablet_replica.cc:442] T 5541f160d628498a8b8ec9fdf572b1b3 P c4f57c71a456480fb2168fabd2513952: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:09:42.539513 2022 tablet_replica.cc:439] T 5541f160d628498a8b8ec9fdf572b1b3 P 4c1669f3864049dbad5d2004b0a37145: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 } } peers { permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 } } peers { permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 } } }
I20260924 07:09:42.539625 2022 tablet_replica.cc:442] T 5541f160d628498a8b8ec9fdf572b1b3 P 4c1669f3864049dbad5d2004b0a37145: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:09:42.539652 2018 tablet_bootstrap.cc:492] T 5541f160d628498a8b8ec9fdf572b1b3 P f1ab124407d1401187fcc2295578d2ad: No bootstrap required, opened a new log
I20260924 07:09:42.539700 2018 ts_tablet_manager.cc:1397] T 5541f160d628498a8b8ec9fdf572b1b3 P f1ab124407d1401187fcc2295578d2ad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:09:42.539847 2018 raft_consensus.cc:348] T 5541f160d628498a8b8ec9fdf572b1b3 P f1ab124407d1401187fcc2295578d2ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 } } peers { permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 } } peers { permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 } }
I20260924 07:09:42.539942 2018 raft_consensus.cc:374] T 5541f160d628498a8b8ec9fdf572b1b3 P f1ab124407d1401187fcc2295578d2ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:42.539975 2018 raft_consensus.cc:729] T 5541f160d628498a8b8ec9fdf572b1b3 P f1ab124407d1401187fcc2295578d2ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1ab124407d1401187fcc2295578d2ad, State: Initialized, Role: FOLLOWER
I20260924 07:09:42.540020 2018 consensus_queue.cc:260] T 5541f160d628498a8b8ec9fdf572b1b3 P f1ab124407d1401187fcc2295578d2ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 } } peers { permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 } } peers { permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 } }
I20260924 07:09:42.540126 2018 ts_tablet_manager.cc:1428] T 5541f160d628498a8b8ec9fdf572b1b3 P f1ab124407d1401187fcc2295578d2ad: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 07:09:42.540189 2024 tablet_replica.cc:439] T 5541f160d628498a8b8ec9fdf572b1b3 P f1ab124407d1401187fcc2295578d2ad: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 } } peers { permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 } } peers { permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 } } }
I20260924 07:09:42.540258 2024 tablet_replica.cc:442] T 5541f160d628498a8b8ec9fdf572b1b3 P f1ab124407d1401187fcc2295578d2ad: This TxnStatusTablet replica's current role is: FOLLOWER
W20260924 07:09:42.590569 1994 tablet.cc:2369] T 5541f160d628498a8b8ec9fdf572b1b3 P 4c1669f3864049dbad5d2004b0a37145: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:09:42.672876 2022 raft_consensus.cc:482] T 5541f160d628498a8b8ec9fdf572b1b3 P 4c1669f3864049dbad5d2004b0a37145 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:09:42.672984 2022 raft_consensus.cc:504] T 5541f160d628498a8b8ec9fdf572b1b3 P 4c1669f3864049dbad5d2004b0a37145 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 } } peers { permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 } } peers { permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 } }
I20260924 07:09:42.673389 2022 leader_election.cc:288] T 5541f160d628498a8b8ec9fdf572b1b3 P 4c1669f3864049dbad5d2004b0a37145 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1ab124407d1401187fcc2295578d2ad (127.0.98.1:46873), c4f57c71a456480fb2168fabd2513952 (127.0.98.2:36799)
I20260924 07:09:42.677868 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.673559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46873 (local address 127.0.98.3:44413)
0924 07:09:42.673627 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:42.673642 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:42.673710 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:42.674111 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:42.674112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:42.674114 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:42.674195 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:42.674197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.675405 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.675406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:42.676812 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:42.676815 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.677552 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.677553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:42.677642 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:42.677646 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:09:42.677714 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 07:09:42.677762 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260924 07:09:42.677873 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.673919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44413 (local address 127.0.98.1:46873)
0924 07:09:42.673962 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:42.673965 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:42.673974 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:09:42.674013 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:42.674014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:42.674024 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:42.674089 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:42.674276 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.675284 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.676903 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.677459 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.677538 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:42.677549 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:42.677731 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 07:09:42.677773 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 07:09:42.678170 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5541f160d628498a8b8ec9fdf572b1b3" candidate_uuid: "4c1669f3864049dbad5d2004b0a37145" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1ab124407d1401187fcc2295578d2ad" is_pre_election: true
I20260924 07:09:42.678253 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.673788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36957 (local address 127.0.98.2:36799)
0924 07:09:42.673845 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:42.673848 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:42.673908 (+ 60us) server_negotiation.cc:407] Connection header received
0924 07:09:42.674007 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:42.674009 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:42.674020 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:42.674089 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:42.674802 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.675579 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.677197 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.677800 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.677871 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:42.677881 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:42.678130 (+ 249us) server_negotiation.cc:299] Negotiation successful
0924 07:09:42.678174 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 07:09:42.678305 1667 raft_consensus.cc:2454] T 5541f160d628498a8b8ec9fdf572b1b3 P f1ab124407d1401187fcc2295578d2ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c1669f3864049dbad5d2004b0a37145 in term 0.
I20260924 07:09:42.678507 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.673713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36799 (local address 127.0.98.3:36957)
0924 07:09:42.673853 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:42.673862 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:42.673935 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:42.674283 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:42.674283 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:42.674285 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:42.674689 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:42.674691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.675690 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.675691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:42.677098 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:42.677101 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.677918 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.677919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:42.678027 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:42.678033 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 07:09:42.678365 (+ 332us) client_negotiation.cc:240] Negotiation successful
0924 07:09:42.678418 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":1,"thread_start_us":86,"threads_started":1}
I20260924 07:09:42.678489 1881 leader_election.cc:302] T 5541f160d628498a8b8ec9fdf572b1b3 P 4c1669f3864049dbad5d2004b0a37145 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c1669f3864049dbad5d2004b0a37145, f1ab124407d1401187fcc2295578d2ad; no voters:
I20260924 07:09:42.678632 2022 raft_consensus.cc:2781] T 5541f160d628498a8b8ec9fdf572b1b3 P 4c1669f3864049dbad5d2004b0a37145 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:09:42.678668 2022 raft_consensus.cc:482] T 5541f160d628498a8b8ec9fdf572b1b3 P 4c1669f3864049dbad5d2004b0a37145 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:09:42.678706 2022 raft_consensus.cc:3037] T 5541f160d628498a8b8ec9fdf572b1b3 P 4c1669f3864049dbad5d2004b0a37145 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:42.678820 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5541f160d628498a8b8ec9fdf572b1b3" candidate_uuid: "4c1669f3864049dbad5d2004b0a37145" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f57c71a456480fb2168fabd2513952" is_pre_election: true
I20260924 07:09:42.678972 1801 raft_consensus.cc:2454] T 5541f160d628498a8b8ec9fdf572b1b3 P c4f57c71a456480fb2168fabd2513952 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c1669f3864049dbad5d2004b0a37145 in term 0.
I20260924 07:09:42.679620 2022 raft_consensus.cc:504] T 5541f160d628498a8b8ec9fdf572b1b3 P 4c1669f3864049dbad5d2004b0a37145 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 } } peers { permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 } } peers { permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 } }
I20260924 07:09:42.679766 2022 leader_election.cc:288] T 5541f160d628498a8b8ec9fdf572b1b3 P 4c1669f3864049dbad5d2004b0a37145 [CANDIDATE]: Term 1 election: Requested vote from peers f1ab124407d1401187fcc2295578d2ad (127.0.98.1:46873), c4f57c71a456480fb2168fabd2513952 (127.0.98.2:36799)
I20260924 07:09:42.679957 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5541f160d628498a8b8ec9fdf572b1b3" candidate_uuid: "4c1669f3864049dbad5d2004b0a37145" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4f57c71a456480fb2168fabd2513952"
I20260924 07:09:42.679958 1667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5541f160d628498a8b8ec9fdf572b1b3" candidate_uuid: "4c1669f3864049dbad5d2004b0a37145" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1ab124407d1401187fcc2295578d2ad"
I20260924 07:09:42.680052 1667 raft_consensus.cc:3037] T 5541f160d628498a8b8ec9fdf572b1b3 P f1ab124407d1401187fcc2295578d2ad [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:42.680055 1801 raft_consensus.cc:3037] T 5541f160d628498a8b8ec9fdf572b1b3 P c4f57c71a456480fb2168fabd2513952 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:42.680805 1667 raft_consensus.cc:2454] T 5541f160d628498a8b8ec9fdf572b1b3 P f1ab124407d1401187fcc2295578d2ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c1669f3864049dbad5d2004b0a37145 in term 1.
I20260924 07:09:42.681002 1881 leader_election.cc:302] T 5541f160d628498a8b8ec9fdf572b1b3 P 4c1669f3864049dbad5d2004b0a37145 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c1669f3864049dbad5d2004b0a37145, f1ab124407d1401187fcc2295578d2ad; no voters:
I20260924 07:09:42.681097 2022 raft_consensus.cc:2781] T 5541f160d628498a8b8ec9fdf572b1b3 P 4c1669f3864049dbad5d2004b0a37145 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:09:42.681113 1801 raft_consensus.cc:2454] T 5541f160d628498a8b8ec9fdf572b1b3 P c4f57c71a456480fb2168fabd2513952 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c1669f3864049dbad5d2004b0a37145 in term 1.
I20260924 07:09:42.681226 2022 raft_consensus.cc:686] T 5541f160d628498a8b8ec9fdf572b1b3 P 4c1669f3864049dbad5d2004b0a37145 [term 1 LEADER]: Becoming Leader. State: Replica: 4c1669f3864049dbad5d2004b0a37145, State: Running, Role: LEADER
I20260924 07:09:42.681327 2022 consensus_queue.cc:237] T 5541f160d628498a8b8ec9fdf572b1b3 P 4c1669f3864049dbad5d2004b0a37145 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 } } peers { permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 } } peers { permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 } }
I20260924 07:09:42.681697 2055 tablet_replica.cc:439] T 5541f160d628498a8b8ec9fdf572b1b3 P 4c1669f3864049dbad5d2004b0a37145: TxnStatusTablet state changed. Reason: New leader 4c1669f3864049dbad5d2004b0a37145. Latest consensus state: current_term: 1 leader_uuid: "4c1669f3864049dbad5d2004b0a37145" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 } } peers { permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 } } peers { permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 } } }
I20260924 07:09:42.681833 2055 tablet_replica.cc:442] T 5541f160d628498a8b8ec9fdf572b1b3 P 4c1669f3864049dbad5d2004b0a37145: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:09:42.682044 2057 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 07:09:42.682175 2057 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 07:09:42.682097 1509 catalog_manager.cc:5495] T 5541f160d628498a8b8ec9fdf572b1b3 P 4c1669f3864049dbad5d2004b0a37145 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c1669f3864049dbad5d2004b0a37145 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "4c1669f3864049dbad5d2004b0a37145" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 } health_report { overall_health: HEALTHY } } }
I20260924 07:09:42.688927 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.685483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41544 (local address 127.0.98.3:43181)
0924 07:09:42.685523 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:42.685525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:42.685535 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:09:42.685600 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:42.685601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:42.685618 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:42.685695 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:09:42.685872 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.686654 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.687885 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.688222 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.688298 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:42.688539 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:09:42.688661 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:09:42.688781 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:42.688832 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 07:09:42.688854 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 07:09:42.688971 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.685401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43181 (local address 127.0.0.1:41544)
0924 07:09:42.685466 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:42.685478 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:42.685529 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:42.685696 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:42.685697 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:42.685699 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:09:42.685795 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:42.685798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.686761 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.686762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:42.687799 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:42.687802 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.688308 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.688309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:42.688387 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:42.688404 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:09:42.688726 (+ 322us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:09:42.688763 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 07:09:42.688839 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 07:09:42.688893 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
W20260924 07:09:42.689754 1854 tablet.cc:2369] T 5541f160d628498a8b8ec9fdf572b1b3 P c4f57c71a456480fb2168fabd2513952: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:09:42.689951 1801 raft_consensus.cc:1261] T 5541f160d628498a8b8ec9fdf572b1b3 P c4f57c71a456480fb2168fabd2513952 [term 1 FOLLOWER]: Refusing update from remote peer 4c1669f3864049dbad5d2004b0a37145: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:09:42.689957 1667 raft_consensus.cc:1261] T 5541f160d628498a8b8ec9fdf572b1b3 P f1ab124407d1401187fcc2295578d2ad [term 1 FOLLOWER]: Refusing update from remote peer 4c1669f3864049dbad5d2004b0a37145: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:09:42.690192 2022 consensus_queue.cc:1035] T 5541f160d628498a8b8ec9fdf572b1b3 P 4c1669f3864049dbad5d2004b0a37145 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:09:42.690349 2060 consensus_queue.cc:1035] T 5541f160d628498a8b8ec9fdf572b1b3 P 4c1669f3864049dbad5d2004b0a37145 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:09:42.690936 2023 tablet_replica.cc:439] T 5541f160d628498a8b8ec9fdf572b1b3 P c4f57c71a456480fb2168fabd2513952: TxnStatusTablet state changed. Reason: New leader 4c1669f3864049dbad5d2004b0a37145. Latest consensus state: current_term: 1 leader_uuid: "4c1669f3864049dbad5d2004b0a37145" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 } } peers { permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 } } peers { permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 } } }
I20260924 07:09:42.691025 2023 tablet_replica.cc:442] T 5541f160d628498a8b8ec9fdf572b1b3 P c4f57c71a456480fb2168fabd2513952: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:09:42.691008 2024 tablet_replica.cc:439] T 5541f160d628498a8b8ec9fdf572b1b3 P f1ab124407d1401187fcc2295578d2ad: TxnStatusTablet state changed. Reason: New leader 4c1669f3864049dbad5d2004b0a37145. Latest consensus state: current_term: 1 leader_uuid: "4c1669f3864049dbad5d2004b0a37145" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 } } peers { permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 } } peers { permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 } } }
I20260924 07:09:42.691108 2024 tablet_replica.cc:442] T 5541f160d628498a8b8ec9fdf572b1b3 P f1ab124407d1401187fcc2295578d2ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:09:42.692004 2023 tablet_replica.cc:439] T 5541f160d628498a8b8ec9fdf572b1b3 P c4f57c71a456480fb2168fabd2513952: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c1669f3864049dbad5d2004b0a37145" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 } } peers { permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 } } peers { permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 } } }
I20260924 07:09:42.692082 2023 tablet_replica.cc:442] T 5541f160d628498a8b8ec9fdf572b1b3 P c4f57c71a456480fb2168fabd2513952: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:09:42.692060 2031 mvcc.cc:205] Tried to move back new op lower bound from 7332797573896138752 to 7332797573863432192. Current Snapshot: MvccSnapshot[applied={T|T < 7332797573896138752}]
I20260924 07:09:42.692203 2060 tablet_replica.cc:439] T 5541f160d628498a8b8ec9fdf572b1b3 P 4c1669f3864049dbad5d2004b0a37145: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c1669f3864049dbad5d2004b0a37145" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 } } peers { permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 } } peers { permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 } } }
I20260924 07:09:42.692277 2060 tablet_replica.cc:442] T 5541f160d628498a8b8ec9fdf572b1b3 P 4c1669f3864049dbad5d2004b0a37145: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:09:42.692334 2060 tablet_replica.cc:439] T 5541f160d628498a8b8ec9fdf572b1b3 P 4c1669f3864049dbad5d2004b0a37145: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c1669f3864049dbad5d2004b0a37145" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 } } peers { permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 } } peers { permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 } } }
I20260924 07:09:42.692373 2060 tablet_replica.cc:442] T 5541f160d628498a8b8ec9fdf572b1b3 P 4c1669f3864049dbad5d2004b0a37145: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:09:42.692369 2024 tablet_replica.cc:439] T 5541f160d628498a8b8ec9fdf572b1b3 P f1ab124407d1401187fcc2295578d2ad: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c1669f3864049dbad5d2004b0a37145" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 } } peers { permanent_uuid: "c4f57c71a456480fb2168fabd2513952" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36799 } } peers { permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 } } }
I20260924 07:09:42.692446 2024 tablet_replica.cc:442] T 5541f160d628498a8b8ec9fdf572b1b3 P f1ab124407d1401187fcc2295578d2ad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:09:42.698210 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.696014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53223 (local address 127.0.98.3:43181)
0924 07:09:42.696054 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:42.696056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:42.696065 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:09:42.696123 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:42.696124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:42.696134 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:42.696182 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:09:42.696280 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.696820 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.697533 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.697733 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.697806 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:42.697979 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:09:42.698008 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:09:42.698076 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:42.698134 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 07:09:42.698149 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 07:09:42.698254 1859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.695956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43181 (local address 127.0.98.2:53223)
0924 07:09:42.696013 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:42.696024 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:42.696067 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:42.696190 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:42.696190 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:42.696191 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:09:42.696233 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:42.696234 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.696909 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.696910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:42.697449 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:42.697451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.697821 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.697822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:42.697878 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:42.697890 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:09:42.698076 (+ 186us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:09:42.698079 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:09:42.698130 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 07:09:42.698174 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260924 07:09:42.700426 1947 raft_consensus.cc:1261] T 574ae28683784d6c8db0bf88de2d822f P 4c1669f3864049dbad5d2004b0a37145 [term 1 FOLLOWER]: Refusing update from remote peer c4f57c71a456480fb2168fabd2513952: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:09:42.700444 1667 raft_consensus.cc:1261] T 574ae28683784d6c8db0bf88de2d822f P f1ab124407d1401187fcc2295578d2ad [term 1 FOLLOWER]: Refusing update from remote peer c4f57c71a456480fb2168fabd2513952: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:09:42.700618 2023 consensus_queue.cc:1035] T 574ae28683784d6c8db0bf88de2d822f P c4f57c71a456480fb2168fabd2513952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1ab124407d1401187fcc2295578d2ad" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:09:42.700837 2023 consensus_queue.cc:1035] T 574ae28683784d6c8db0bf88de2d822f P c4f57c71a456480fb2168fabd2513952 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c1669f3864049dbad5d2004b0a37145" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:09:42.708380 1999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.705319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36799 (local address 127.0.98.3:52659)
0924 07:09:42.705413 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:42.705424 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:42.705473 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:42.705613 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:42.705614 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:42.705615 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:09:42.705662 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:42.705662 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.706614 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.706615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:42.707453 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:42.707455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.707985 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.707986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:42.708044 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:42.708056 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:09:42.708256 (+ 200us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:09:42.708257 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 07:09:42.708298 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 07:09:42.708329 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 07:09:42.708400 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.705349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52659 (local address 127.0.98.2:36799)
0924 07:09:42.705393 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:42.705395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:42.705458 (+ 63us) server_negotiation.cc:407] Connection header received
0924 07:09:42.705525 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:42.705526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:42.705538 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:42.705597 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:09:42.705729 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.706520 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.707541 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.707874 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.707954 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:42.708151 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:09:42.708187 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:09:42.708255 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:42.708305 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 07:09:42.708326 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 07:09:42.717907 1509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:09:42.717990 1509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:09:42.718611 1509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5528fa3ab4b145ebbc6cd8f9eb434d15: Sending DeleteTablet for 3 replicas of tablet 574ae28683784d6c8db0bf88de2d822f
I20260924 07:09:42.718881 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 574ae28683784d6c8db0bf88de2d822f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60666
I20260924 07:09:42.718888 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet 574ae28683784d6c8db0bf88de2d822f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55794
I20260924 07:09:42.718976 2038 tablet_replica.cc:330] T 574ae28683784d6c8db0bf88de2d822f P c4f57c71a456480fb2168fabd2513952: stopping tablet replica
I20260924 07:09:42.718979 2040 tablet_replica.cc:330] T 574ae28683784d6c8db0bf88de2d822f P f1ab124407d1401187fcc2295578d2ad: stopping tablet replica
I20260924 07:09:42.719020 2040 raft_consensus.cc:2229] T 574ae28683784d6c8db0bf88de2d822f P f1ab124407d1401187fcc2295578d2ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:09:42.719022 2038 raft_consensus.cc:2229] T 574ae28683784d6c8db0bf88de2d822f P c4f57c71a456480fb2168fabd2513952 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:09:42.719051 2040 raft_consensus.cc:2258] T 574ae28683784d6c8db0bf88de2d822f P f1ab124407d1401187fcc2295578d2ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:09:42.719074 2038 raft_consensus.cc:2258] T 574ae28683784d6c8db0bf88de2d822f P c4f57c71a456480fb2168fabd2513952 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:09:42.719379 2038 ts_tablet_manager.cc:1905] T 574ae28683784d6c8db0bf88de2d822f P c4f57c71a456480fb2168fabd2513952: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:09:42.719381 2040 ts_tablet_manager.cc:1905] T 574ae28683784d6c8db0bf88de2d822f P f1ab124407d1401187fcc2295578d2ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:09:42.719605 1921 tablet_service.cc:1514] Processing DeleteTablet for tablet 574ae28683784d6c8db0bf88de2d822f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41540
I20260924 07:09:42.719697 2039 tablet_replica.cc:330] T 574ae28683784d6c8db0bf88de2d822f P 4c1669f3864049dbad5d2004b0a37145: stopping tablet replica
I20260924 07:09:42.719753 2039 raft_consensus.cc:2229] T 574ae28683784d6c8db0bf88de2d822f P 4c1669f3864049dbad5d2004b0a37145 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:09:42.719794 2039 raft_consensus.cc:2258] T 574ae28683784d6c8db0bf88de2d822f P 4c1669f3864049dbad5d2004b0a37145 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:09:42.720088 2039 ts_tablet_manager.cc:1905] T 574ae28683784d6c8db0bf88de2d822f P 4c1669f3864049dbad5d2004b0a37145: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:09:42.721551 2038 ts_tablet_manager.cc:1918] T 574ae28683784d6c8db0bf88de2d822f P c4f57c71a456480fb2168fabd2513952: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:09:42.721555 2040 ts_tablet_manager.cc:1918] T 574ae28683784d6c8db0bf88de2d822f P f1ab124407d1401187fcc2295578d2ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:09:42.721617 2038 log.cc:1196] T 574ae28683784d6c8db0bf88de2d822f P c4f57c71a456480fb2168fabd2513952: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-1/wal/wals/574ae28683784d6c8db0bf88de2d822f
I20260924 07:09:42.721617 2040 log.cc:1196] T 574ae28683784d6c8db0bf88de2d822f P f1ab124407d1401187fcc2295578d2ad: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-0/wal/wals/574ae28683784d6c8db0bf88de2d822f
I20260924 07:09:42.721843 2038 ts_tablet_manager.cc:1939] T 574ae28683784d6c8db0bf88de2d822f P c4f57c71a456480fb2168fabd2513952: Deleting consensus metadata
I20260924 07:09:42.721858 2040 ts_tablet_manager.cc:1939] T 574ae28683784d6c8db0bf88de2d822f P f1ab124407d1401187fcc2295578d2ad: Deleting consensus metadata
I20260924 07:09:42.721911 2039 ts_tablet_manager.cc:1918] T 574ae28683784d6c8db0bf88de2d822f P 4c1669f3864049dbad5d2004b0a37145: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:09:42.721974 2039 log.cc:1196] T 574ae28683784d6c8db0bf88de2d822f P 4c1669f3864049dbad5d2004b0a37145: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790233772407728-392-0/minicluster-data/ts-2/wal/wals/574ae28683784d6c8db0bf88de2d822f
I20260924 07:09:42.722144 1497 catalog_manager.cc:4841] TS f1ab124407d1401187fcc2295578d2ad (127.0.98.1:46873): tablet 574ae28683784d6c8db0bf88de2d822f (table test-table [id=575a4a05926a4f178768056259e3e386]) successfully deleted
I20260924 07:09:42.722189 2039 ts_tablet_manager.cc:1939] T 574ae28683784d6c8db0bf88de2d822f P 4c1669f3864049dbad5d2004b0a37145: Deleting consensus metadata
I20260924 07:09:42.722229 1497 catalog_manager.cc:4841] TS c4f57c71a456480fb2168fabd2513952 (127.0.98.2:36799): tablet 574ae28683784d6c8db0bf88de2d822f (table test-table [id=575a4a05926a4f178768056259e3e386]) successfully deleted
I20260924 07:09:42.722586 1494 catalog_manager.cc:4841] TS 4c1669f3864049dbad5d2004b0a37145 (127.0.98.3:43181): tablet 574ae28683784d6c8db0bf88de2d822f (table test-table [id=575a4a05926a4f178768056259e3e386]) successfully deleted
Sep 24 07:09:42 dist-test-slave-14ww krb5kdc[1465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233782, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 07:09:42.733346 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.722537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55802 (local address 127.0.98.1:46873)
0924 07:09:42.722578 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:42.722580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:42.722818 (+ 238us) server_negotiation.cc:407] Connection header received
0924 07:09:42.722843 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:42.722843 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:42.722856 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:42.722916 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:42.723440 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.724168 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.724959 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.725279 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.725348 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:42.729231 (+ 3883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:42.729239 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:42.729239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:42.729247 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:42.732147 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:42.732511 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:42.732515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:42.732515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:42.732578 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:42.732817 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:42.732818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:42.732818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:42.732979 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:42.733115 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:42.733164 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 07:09:42.733239 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 07:09:42.733632 1707 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55802
I20260924 07:09:42.742230 1559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:42.734565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36178 (local address 127.0.98.62:35085)
0924 07:09:42.734606 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:42.734608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:42.734669 (+ 61us) server_negotiation.cc:407] Connection header received
0924 07:09:42.734732 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:42.734733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:42.734743 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:42.734798 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:42.735238 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.736002 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.736666 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:42.736989 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:42.737059 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:42.738836 (+ 1777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:42.738837 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:42.738838 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:42.738844 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:42.741201 (+ 2357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:42.741523 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:42.741524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:42.741524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:42.741583 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:42.741776 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:42.741777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:42.741778 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:42.741899 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:42.741992 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:42.742065 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 07:09:42.742135 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 07:09:42.742468 1509 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36178
I20260924 07:09:42.743018 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 1582
I20260924 07:09:42.748895 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 1722
I20260924 07:09:42.756110 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 1862
I20260924 07:09:42.761600 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 1478
2026-09-24T07:09:42Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5334 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:09:42 dist-test-slave-14ww krb5kdc[2077](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:09:42 dist-test-slave-14ww krb5kdc[2077](info): set up 2 sockets
Sep 24 07:09:42 dist-test-slave-14ww krb5kdc[2077](info): commencing operation
krb5kdc: starting...
W20260924 07:09:43.800702 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:09:43 dist-test-slave-14ww krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:09:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:09:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:09:43.919184 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37287
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37287
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:43.999466 2090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:43.999656 2090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:43.999679 2090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:44.000907 2090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:09:44.000947 2090 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 07:09:44.000960 2090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:44.000973 2090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:44.001022 2090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:09:44.001039 2090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:09:44.002504 2090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37287
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:44.002740 2090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:44.002960 2090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:44.007952 2096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:44.007999 2098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:44.007979 2095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:44.008258 2090 server_base.cc:1034] running on GCE node
I20260924 07:09:44.008831 2090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:44.009155 2090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:44.010313 2090 hybrid_clock.cc:648] HybridClock initialized: now 1790233784010300 us; error 32 us; skew 500 ppm
Sep 24 07:09:44 dist-test-slave-14ww krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:44.013294 2090 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:09:44.013751 2090 webserver.cc:460] Webserver started at http://127.0.98.62:44721/ using document root <none> and password file <none>
I20260924 07:09:44.013970 2090 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:44.014036 2090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:44.014164 2090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:44.015178 2090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "5283f28bbbbf47789cdb632a6923f1ba"
format_stamp: "Formatted at 2026-09-24 07:09:44 on dist-test-slave-14ww"
server_key: "fb33dbc812f1f95b278848b1f94fdb43"
server_key_iv: "daa59c2c071b5bdf1a84eb15112034f2"
server_key_version: "encryptionkey@0"
I20260924 07:09:44.015547 2090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "5283f28bbbbf47789cdb632a6923f1ba"
format_stamp: "Formatted at 2026-09-24 07:09:44 on dist-test-slave-14ww"
server_key: "fb33dbc812f1f95b278848b1f94fdb43"
server_key_iv: "daa59c2c071b5bdf1a84eb15112034f2"
server_key_version: "encryptionkey@0"
I20260924 07:09:44.016917 2090 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 07:09:44.017709 2105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:44.017889 2090 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:09:44.017990 2090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "5283f28bbbbf47789cdb632a6923f1ba"
format_stamp: "Formatted at 2026-09-24 07:09:44 on dist-test-slave-14ww"
server_key: "fb33dbc812f1f95b278848b1f94fdb43"
server_key_iv: "daa59c2c071b5bdf1a84eb15112034f2"
server_key_version: "encryptionkey@0"
I20260924 07:09:44.018074 2090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:44.026669 2090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:44.029990 2090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:44.030149 2090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:44.034906 2090 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37287
I20260924 07:09:44.034950 2167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37287 every 8 connection(s)
I20260924 07:09:44.035288 2090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:09:44.035859 2168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:44.038144 2168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5283f28bbbbf47789cdb632a6923f1ba: Bootstrap starting.
I20260924 07:09:44.038725 2168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5283f28bbbbf47789cdb632a6923f1ba: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:44.039000 2168 log.cc:824] T 00000000000000000000000000000000 P 5283f28bbbbf47789cdb632a6923f1ba: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:44.039676 2168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5283f28bbbbf47789cdb632a6923f1ba: No bootstrap required, opened a new log
I20260924 07:09:44.041786 2168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5283f28bbbbf47789cdb632a6923f1ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5283f28bbbbf47789cdb632a6923f1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37287 } }
I20260924 07:09:44.041916 2168 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5283f28bbbbf47789cdb632a6923f1ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:44.041937 2168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5283f28bbbbf47789cdb632a6923f1ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5283f28bbbbf47789cdb632a6923f1ba, State: Initialized, Role: FOLLOWER
I20260924 07:09:44.042101 2168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5283f28bbbbf47789cdb632a6923f1ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5283f28bbbbf47789cdb632a6923f1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37287 } }
I20260924 07:09:44.042160 2168 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5283f28bbbbf47789cdb632a6923f1ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:09:44.042200 2168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5283f28bbbbf47789cdb632a6923f1ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:09:44.042245 2168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5283f28bbbbf47789cdb632a6923f1ba [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:44.042747 2168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5283f28bbbbf47789cdb632a6923f1ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5283f28bbbbf47789cdb632a6923f1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37287 } }
I20260924 07:09:44.042851 2168 leader_election.cc:302] T 00000000000000000000000000000000 P 5283f28bbbbf47789cdb632a6923f1ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5283f28bbbbf47789cdb632a6923f1ba; no voters:
I20260924 07:09:44.043033 2168 leader_election.cc:288] T 00000000000000000000000000000000 P 5283f28bbbbf47789cdb632a6923f1ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:09:44.043095 2171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5283f28bbbbf47789cdb632a6923f1ba [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:09:44.043088 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 2090
I20260924 07:09:44.043187 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:09:44.043247 2171 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5283f28bbbbf47789cdb632a6923f1ba [term 1 LEADER]: Becoming Leader. State: Replica: 5283f28bbbbf47789cdb632a6923f1ba, State: Running, Role: LEADER
I20260924 07:09:44.043287 2168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5283f28bbbbf47789cdb632a6923f1ba [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:09:44.043373 392 external_mini_cluster.cc:1428] Setting key d119f1e238dbd3710da2629bd365f169
I20260924 07:09:44.043380 2171 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5283f28bbbbf47789cdb632a6923f1ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5283f28bbbbf47789cdb632a6923f1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37287 } }
I20260924 07:09:44.043754 2173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5283f28bbbbf47789cdb632a6923f1ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5283f28bbbbf47789cdb632a6923f1ba. Latest consensus state: current_term: 1 leader_uuid: "5283f28bbbbf47789cdb632a6923f1ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5283f28bbbbf47789cdb632a6923f1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37287 } } }
I20260924 07:09:44.043833 2173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5283f28bbbbf47789cdb632a6923f1ba [sys.catalog]: This master's current role is: LEADER
I20260924 07:09:44.044044 2179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:09:44.044572 2172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5283f28bbbbf47789cdb632a6923f1ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5283f28bbbbf47789cdb632a6923f1ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5283f28bbbbf47789cdb632a6923f1ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37287 } } }
I20260924 07:09:44.044723 2172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5283f28bbbbf47789cdb632a6923f1ba [sys.catalog]: This master's current role is: LEADER
I20260924 07:09:44.044866 2179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:09:44.046446 2179 catalog_manager.cc:1347] Generated new cluster ID: 98265fc618604520ad218d353a19fd03
I20260924 07:09:44.046502 2179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 07:09:44 dist-test-slave-14ww krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:09:44.056864 2182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:44.044006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39722 (local address 127.0.98.62:37287)
0924 07:09:44.044506 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:44.044516 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:44.044549 (+ 33us) server_negotiation.cc:407] Connection header received
0924 07:09:44.044667 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:44.044674 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:44.044726 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:44.044905 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:44.045578 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.046908 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.047926 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.048550 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.048662 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:44.052003 (+ 3341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:44.052007 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:44.052008 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:44.052019 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:44.055229 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:44.055614 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:44.055617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:44.055617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:44.055720 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:44.055920 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:44.055921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:44.055922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:44.056222 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:44.056387 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:44.056463 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 07:09:44.056596 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":62,"threads_started":1}
I20260924 07:09:44.064675 2179 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:09:44.065178 2179 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:09:44.067834 2179 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5283f28bbbbf47789cdb632a6923f1ba: Generated new TSK 0
I20260924 07:09:44.068008 2179 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 07:09:44.099793 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37287
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:44.186208 2194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:44.186406 2194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:44.186430 2194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:44.187538 2194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:44.187577 2194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:44.187597 2194 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 07:09:44.187609 2194 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:09:44.187625 2194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:09:44.189054 2194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:44.189308 2194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:44.189531 2194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:44.194583 2202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:44.194567 2199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:44.194628 2194 server_base.cc:1034] running on GCE node
W20260924 07:09:44.194566 2200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:44.194960 2194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:44.195227 2194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:44.196395 2194 hybrid_clock.cc:648] HybridClock initialized: now 1790233784196370 us; error 39 us; skew 500 ppm
Sep 24 07:09:44 dist-test-slave-14ww krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:44.199437 2194 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:09:44.199910 2194 webserver.cc:460] Webserver started at http://127.0.98.1:45355/ using document root <none> and password file <none>
I20260924 07:09:44.200150 2194 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:44.200218 2194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:44.200357 2194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:44.201445 2194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-0/data/instance:
uuid: "ae93f97d901b4fa0aba5531618b11c0c"
format_stamp: "Formatted at 2026-09-24 07:09:44 on dist-test-slave-14ww"
server_key: "887b66b775e47b07bed76b69a39f27f2"
server_key_iv: "3068a5a33af65d89f3f6edec64f217f9"
server_key_version: "encryptionkey@0"
I20260924 07:09:44.201838 2194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ae93f97d901b4fa0aba5531618b11c0c"
format_stamp: "Formatted at 2026-09-24 07:09:44 on dist-test-slave-14ww"
server_key: "887b66b775e47b07bed76b69a39f27f2"
server_key_iv: "3068a5a33af65d89f3f6edec64f217f9"
server_key_version: "encryptionkey@0"
I20260924 07:09:44.203123 2194 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:09:44.203852 2209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:44.204073 2194 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:09:44.204169 2194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-0/wal
uuid: "ae93f97d901b4fa0aba5531618b11c0c"
format_stamp: "Formatted at 2026-09-24 07:09:44 on dist-test-slave-14ww"
server_key: "887b66b775e47b07bed76b69a39f27f2"
server_key_iv: "3068a5a33af65d89f3f6edec64f217f9"
server_key_version: "encryptionkey@0"
I20260924 07:09:44.204244 2194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:44.218571 2194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:44.222009 2194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:44.222117 2194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:44.222541 2194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:44.222607 2194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:44.222651 2194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:44.222678 2194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:44.229597 2194 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35995
I20260924 07:09:44.229676 2324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35995 every 8 connection(s)
I20260924 07:09:44.230002 2194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 07:09:44 dist-test-slave-14ww krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:09:44.233997 2182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:44.223090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38807 (local address 127.0.98.62:37287)
0924 07:09:44.223130 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:44.223132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:44.223377 (+ 245us) server_negotiation.cc:407] Connection header received
0924 07:09:44.223884 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:44.223890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:44.223903 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:44.223954 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:44.224795 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.225599 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.227182 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.227506 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.227585 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:44.230895 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:44.230896 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:44.230897 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:44.230903 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:44.232912 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:44.233265 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:44.233265 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:44.233266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:44.233313 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:44.233500 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:44.233500 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:44.233501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:44.233594 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:44.233644 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:44.233877 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 07:09:44.233934 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 07:09:44.234182 2218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:44.223090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37287 (local address 127.0.98.1:38807)
0924 07:09:44.223267 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:44.223299 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:44.223773 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:44.224029 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:44.224033 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:44.224361 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:44.224644 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:44.224654 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.225721 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.225721 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:44.227108 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:44.227111 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.227616 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.227617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:44.227776 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:44.229088 (+ 1312us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:44.229097 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:44.230815 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:44.232982 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:44.232988 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:44.232989 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:44.233197 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:44.233357 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:44.233358 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:44.233358 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:44.233445 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:44.233651 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:44.233655 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:44.233769 (+ 114us) client_negotiation.cc:769] Sending connection context
0924 07:09:44.233849 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 07:09:44.233961 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":113,"threads_started":1}
I20260924 07:09:44.234651 2325 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37287
I20260924 07:09:44.234731 2325 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:44.234905 2325 heartbeater.cc:511] Master 127.0.98.62:37287 requested a full tablet report, sending...
I20260924 07:09:44.234941 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 2194
I20260924 07:09:44.235028 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:09:44.235225 392 external_mini_cluster.cc:1428] Setting key a2514c9d5fce512d94fd414389b50dd8
I20260924 07:09:44.235402 2122 ts_manager.cc:194] Registered new tserver with Master: ae93f97d901b4fa0aba5531618b11c0c (127.0.98.1:35995)
I20260924 07:09:44.236339 2122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38807
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 07:09:44.244859 2182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:44.236121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41557 (local address 127.0.98.62:37287)
0924 07:09:44.236534 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:44.236536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:44.236675 (+ 139us) server_negotiation.cc:407] Connection header received
0924 07:09:44.236788 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:44.236788 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:44.236798 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:44.236828 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:44.237207 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.237777 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.238883 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.239059 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.239126 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:44.240830 (+ 1704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:44.240832 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:44.240832 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:44.240838 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:44.243672 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:44.244060 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:44.244061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:44.244062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:44.244117 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:44.244321 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:44.244322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:44.244323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:44.244444 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:44.244520 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:44.244683 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 07:09:44.244792 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393}
I20260924 07:09:44.244885 2331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:44.236040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37287 (local address 127.0.98.1:41557)
0924 07:09:44.236603 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:44.236613 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:44.236689 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:44.236883 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:44.236884 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:44.237066 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:44.237121 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:44.237123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.237911 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.237912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:44.238796 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:44.238799 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.239147 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.239148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:44.239229 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:44.240274 (+ 1045us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:44.240280 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:44.240699 (+ 419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:44.243781 (+ 3082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:44.243782 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:44.243783 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:44.243954 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:44.244191 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:44.244192 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:44.244192 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:44.244249 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:44.244546 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:44.244547 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:44.244590 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 07:09:44.244641 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 07:09:44.244714 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":67,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 07:09:44.259896 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37287
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:44.339639 2334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:44.339835 2334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:44.339860 2334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:44.340996 2334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:44.341034 2334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:44.341054 2334 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 07:09:44.341066 2334 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:09:44.341121 2334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:09:44.342492 2334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:44.342716 2334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:44.342979 2334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:44.348131 2339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:44.348124 2342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:44.348109 2340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:44.348325 2334 server_base.cc:1034] running on GCE node
I20260924 07:09:44.348590 2334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:44.348869 2334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:44.350051 2334 hybrid_clock.cc:648] HybridClock initialized: now 1790233784350035 us; error 44 us; skew 500 ppm
Sep 24 07:09:44 dist-test-slave-14ww krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:44.353183 2334 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:09:44.353602 2334 webserver.cc:460] Webserver started at http://127.0.98.2:35199/ using document root <none> and password file <none>
I20260924 07:09:44.353832 2334 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:44.353896 2334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:44.354033 2334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:44.355067 2334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-1/data/instance:
uuid: "9afadad6d73540b88766e90a1684b61f"
format_stamp: "Formatted at 2026-09-24 07:09:44 on dist-test-slave-14ww"
server_key: "24256bc6c71d3392388eddb4236ab5fc"
server_key_iv: "8f107e365f6c2b81fd6f2e2674bba641"
server_key_version: "encryptionkey@0"
I20260924 07:09:44.355428 2334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9afadad6d73540b88766e90a1684b61f"
format_stamp: "Formatted at 2026-09-24 07:09:44 on dist-test-slave-14ww"
server_key: "24256bc6c71d3392388eddb4236ab5fc"
server_key_iv: "8f107e365f6c2b81fd6f2e2674bba641"
server_key_version: "encryptionkey@0"
I20260924 07:09:44.356719 2334 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:09:44.357460 2349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:44.357633 2334 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:09:44.357724 2334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-1/wal
uuid: "9afadad6d73540b88766e90a1684b61f"
format_stamp: "Formatted at 2026-09-24 07:09:44 on dist-test-slave-14ww"
server_key: "24256bc6c71d3392388eddb4236ab5fc"
server_key_iv: "8f107e365f6c2b81fd6f2e2674bba641"
server_key_version: "encryptionkey@0"
I20260924 07:09:44.357817 2334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:44.374009 2334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:44.377038 2334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:44.377185 2334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:44.377641 2334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:44.377702 2334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:44.377748 2334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:44.377786 2334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:44.384548 2334 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45009
I20260924 07:09:44.384629 2464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45009 every 8 connection(s)
I20260924 07:09:44.385006 2334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 07:09:44 dist-test-slave-14ww krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:09:44.389093 2182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:44.378141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39895 (local address 127.0.98.62:37287)
0924 07:09:44.378200 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:44.378202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:44.378427 (+ 225us) server_negotiation.cc:407] Connection header received
0924 07:09:44.378908 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:44.378910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:44.378921 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:44.378988 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:44.379742 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.380550 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.382248 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.382555 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.382620 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:44.385971 (+ 3351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:44.385972 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:44.385973 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:44.385979 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:44.387903 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:44.388254 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:44.388254 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:44.388255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:44.388302 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:44.388511 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:44.388511 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:44.388512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:44.388610 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:44.388659 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:44.388959 (+ 300us) server_negotiation.cc:299] Negotiation successful
0924 07:09:44.389028 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 07:09:44.389223 2358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:44.378128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37287 (local address 127.0.98.2:39895)
0924 07:09:44.378316 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:44.378348 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:44.378809 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:44.379053 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:44.379057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:44.379389 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:44.379666 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:44.379671 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.380655 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.380656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:44.382155 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:44.382159 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.382644 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.382645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:44.382802 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:44.384051 (+ 1249us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:44.384060 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:44.385874 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:44.387978 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:44.387983 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:44.387984 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:44.388189 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:44.388361 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:44.388362 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:44.388362 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:44.388453 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:44.388665 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:44.388669 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:44.388848 (+ 179us) client_negotiation.cc:769] Sending connection context
0924 07:09:44.388932 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 07:09:44.389046 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":96,"threads_started":1}
I20260924 07:09:44.389642 2465 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37287
I20260924 07:09:44.389724 2465 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:44.389899 2465 heartbeater.cc:511] Master 127.0.98.62:37287 requested a full tablet report, sending...
I20260924 07:09:44.390239 2121 ts_manager.cc:194] Registered new tserver with Master: 9afadad6d73540b88766e90a1684b61f (127.0.98.2:45009)
I20260924 07:09:44.390758 2121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39895
I20260924 07:09:44.394362 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 2334
I20260924 07:09:44.394482 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:09:44.394690 392 external_mini_cluster.cc:1428] Setting key 0e0f41eced3719b812a4f79e09409fd6
I20260924 07:09:44.397848 2182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:44.390685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39035 (local address 127.0.98.62:37287)
0924 07:09:44.390742 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:44.390744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:44.390884 (+ 140us) server_negotiation.cc:407] Connection header received
0924 07:09:44.390993 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:44.390994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:44.391005 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:44.391084 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:44.391361 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.391863 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.392531 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.392697 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.392861 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:44.394031 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:44.394032 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:44.394033 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:44.394038 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:44.396696 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:44.397082 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:44.397083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:44.397083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:44.397135 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:44.397331 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:44.397332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:44.397332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:44.397472 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:44.397541 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:44.397693 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 07:09:44.397762 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 07:09:44.397872 2471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:44.390636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37287 (local address 127.0.98.2:39035)
0924 07:09:44.390806 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:44.390835 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:44.390907 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:44.391089 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:44.391089 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:44.391244 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:44.391291 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:44.391292 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.391943 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.391944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:44.392463 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:44.392465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.392870 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.392871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:44.392922 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:44.393696 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:44.393700 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:44.393945 (+ 245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:44.396824 (+ 2879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:44.396826 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:44.396826 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:44.396997 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:44.397204 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:44.397205 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:44.397205 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:44.397260 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:44.397543 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:44.397544 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:44.397586 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 07:09:44.397669 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 07:09:44.397762 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":67,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 07:09:44.419199 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37287
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:44.498746 2474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:44.498979 2474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:44.499020 2474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:44.500174 2474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:44.500241 2474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:44.500276 2474 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 07:09:44.500313 2474 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:09:44.500346 2474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:09:44.501878 2474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:37287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:44.502125 2474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:44.502372 2474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:44.507304 2480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:44.507319 2482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:44.507321 2479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:44.507521 2474 server_base.cc:1034] running on GCE node
I20260924 07:09:44.507735 2474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:44.507977 2474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:44.509141 2474 hybrid_clock.cc:648] HybridClock initialized: now 1790233784509132 us; error 54 us; skew 500 ppm
Sep 24 07:09:44 dist-test-slave-14ww krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:44.512277 2474 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:09:44.512805 2474 webserver.cc:460] Webserver started at http://127.0.98.3:46337/ using document root <none> and password file <none>
I20260924 07:09:44.513020 2474 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:44.513087 2474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:44.513219 2474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:44.514221 2474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-2/data/instance:
uuid: "a5cc149836284c468c0bff3680e18479"
format_stamp: "Formatted at 2026-09-24 07:09:44 on dist-test-slave-14ww"
server_key: "5033f6ea7ea3fabeee590571e62f6cb5"
server_key_iv: "031461fb8a96d4667ccd342a112b26e9"
server_key_version: "encryptionkey@0"
I20260924 07:09:44.514581 2474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a5cc149836284c468c0bff3680e18479"
format_stamp: "Formatted at 2026-09-24 07:09:44 on dist-test-slave-14ww"
server_key: "5033f6ea7ea3fabeee590571e62f6cb5"
server_key_iv: "031461fb8a96d4667ccd342a112b26e9"
server_key_version: "encryptionkey@0"
I20260924 07:09:44.515870 2474 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:09:44.516562 2489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:44.516722 2474 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:09:44.516867 2474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-2/wal
uuid: "a5cc149836284c468c0bff3680e18479"
format_stamp: "Formatted at 2026-09-24 07:09:44 on dist-test-slave-14ww"
server_key: "5033f6ea7ea3fabeee590571e62f6cb5"
server_key_iv: "031461fb8a96d4667ccd342a112b26e9"
server_key_version: "encryptionkey@0"
I20260924 07:09:44.516954 2474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:44.542469 2474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:44.546010 2474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:44.546165 2474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:44.546640 2474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:44.546703 2474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:44.546747 2474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:44.546772 2474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 07:09:44 dist-test-slave-14ww krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:09:44.553758 2474 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37165
I20260924 07:09:44.553857 2604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37165 every 8 connection(s)
I20260924 07:09:44.554136 2474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:09:44.557893 2182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:44.547128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51891 (local address 127.0.98.62:37287)
0924 07:09:44.547169 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:44.547171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:44.547397 (+ 226us) server_negotiation.cc:407] Connection header received
0924 07:09:44.547912 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:44.547913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:44.547924 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:44.547993 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:44.548814 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.549638 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.550633 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.550837 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.550923 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:44.553753 (+ 2830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:44.553754 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:44.553755 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:44.553762 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:44.556701 (+ 2939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:44.557202 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:44.557203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:44.557203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:44.557252 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:44.557408 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:44.557409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:44.557409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:44.557494 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:44.557538 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:44.557763 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 07:09:44.557820 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 07:09:44.558027 2498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:44.547127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37287 (local address 127.0.98.3:51891)
0924 07:09:44.547287 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:44.547319 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:44.547811 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:44.548061 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:44.548065 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:44.548407 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:44.548693 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:44.548698 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.549734 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.549735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:44.550538 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:44.550540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.550942 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.550944 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:44.551065 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:44.552262 (+ 1197us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:44.552270 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:44.553642 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:44.556828 (+ 3186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:44.556834 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:44.556836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:44.557139 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:44.557292 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:44.557293 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:44.557293 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:44.557374 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:44.557545 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:44.557550 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:44.557652 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 07:09:44.557731 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 07:09:44.557839 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260924 07:09:44.558516 2605 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37287
I20260924 07:09:44.558594 2605 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:44.558773 2605 heartbeater.cc:511] Master 127.0.98.62:37287 requested a full tablet report, sending...
I20260924 07:09:44.559141 2122 ts_manager.cc:194] Registered new tserver with Master: a5cc149836284c468c0bff3680e18479 (127.0.98.3:37165)
I20260924 07:09:44.559623 2122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51891
I20260924 07:09:44.563505 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 2474
I20260924 07:09:44.563599 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:09:44.563763 392 external_mini_cluster.cc:1428] Setting key 7a19dcc05489d094c4732f5bcc05469f
I20260924 07:09:44.564633 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:09:44.566512 2182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:44.559570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57707 (local address 127.0.98.62:37287)
0924 07:09:44.559607 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:44.559609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:44.559754 (+ 145us) server_negotiation.cc:407] Connection header received
0924 07:09:44.559846 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:44.559847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:44.559857 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:44.559918 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:44.560179 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.560705 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.561500 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.561671 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.561723 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:44.562854 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:44.562855 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:44.562855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:44.562862 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:44.565394 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:44.565775 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:44.565776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:44.565777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:44.565820 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:44.565995 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:44.565995 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:44.565995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:44.566079 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:44.566127 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:44.566281 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 07:09:44.566399 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 07:09:44.566524 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:44.559513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37287 (local address 127.0.98.3:57707)
0924 07:09:44.559693 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:44.559704 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:44.559777 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:44.559919 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:44.559920 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:44.560065 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:44.560111 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:44.560112 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.560835 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.560836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:44.561432 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:44.561434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.561733 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.561733 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:44.561777 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:44.562542 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:44.562546 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:44.562782 (+ 236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:44.565477 (+ 2695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:44.565478 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:44.565478 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:44.565705 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:44.565876 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:44.565877 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:44.565877 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:44.565938 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:44.566141 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:44.566141 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:44.566185 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 07:09:44.566266 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 07:09:44.566378 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":73,"threads_started":1}
Sep 24 07:09:44 dist-test-slave-14ww krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 07:09:44 dist-test-slave-14ww krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:09:44.587975 2182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:44.580227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39732 (local address 127.0.98.62:37287)
0924 07:09:44.580284 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:44.580286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:44.580293 (+ 7us) server_negotiation.cc:407] Connection header received
0924 07:09:44.580325 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:44.580326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:44.580333 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:44.580383 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:44.580960 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.581417 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.582098 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.582269 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.582328 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:44.585092 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:44.585093 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:44.585093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:44.585099 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:44.587102 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:44.587418 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:44.587419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:44.587420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:44.587467 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:44.587623 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:44.587623 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:44.587623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:44.587708 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:44.587776 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:44.587862 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 07:09:44.587914 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 07:09:44.590859 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39732:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:09:44.592504 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:09:44.598726 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:44.594672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37165 (local address 127.0.0.1:53366)
0924 07:09:44.595116 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:44.595126 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:44.595180 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:44.595446 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:44.595446 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:44.595448 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:44.595503 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:44.595505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.596706 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.596707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:44.597644 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:44.597647 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.598440 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.598441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:44.598514 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:44.598527 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 07:09:44.598600 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 07:09:44.598646 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":76,"thread_start_us":81,"threads_started":1}
I20260924 07:09:44.598757 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:44.594762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53366 (local address 127.0.98.3:37165)
0924 07:09:44.595108 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:44.595111 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:44.595176 (+ 65us) server_negotiation.cc:407] Connection header received
0924 07:09:44.595265 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:44.595267 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:44.595281 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:44.595398 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:44.595573 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.596616 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.597747 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.598361 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.598434 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:44.598494 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:44.598625 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 07:09:44.598661 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":72,"threads_started":1}
I20260924 07:09:44.599220 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:44.594634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35995 (local address 127.0.0.1:40000)
0924 07:09:44.594802 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:44.594839 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:44.594910 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:44.595264 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:44.595268 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:44.595270 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:44.595337 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:44.595338 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.596692 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.596693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:44.598004 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:44.598007 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.598967 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.598979 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:44.599049 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:44.599053 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:09:44.599102 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 07:09:44.599147 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":27,"thread_start_us":46,"threads_started":1}
I20260924 07:09:44.599421 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:44.594913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43970 (local address 127.0.98.2:45009)
0924 07:09:44.595076 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:44.595081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:44.595342 (+ 261us) server_negotiation.cc:407] Connection header received
0924 07:09:44.595432 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:44.595434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:44.595449 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:44.595653 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:44.595842 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.596597 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.598294 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.599044 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.599113 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:44.599165 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:44.599252 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 07:09:44.599292 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":93,"threads_started":1}
I20260924 07:09:44.599445 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:44.594959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40000 (local address 127.0.98.1:35995)
0924 07:09:44.595091 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:44.595096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:44.595107 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:09:44.595149 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:44.595151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:44.595167 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:44.595244 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:44.595426 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.596591 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.598092 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.598821 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.598884 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:44.598934 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:44.599312 (+ 378us) server_negotiation.cc:299] Negotiation successful
0924 07:09:44.599351 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":71,"threads_started":1}
I20260924 07:09:44.599556 2539 tablet_service.cc:1467] Processing CreateTablet for tablet 3fc4ceb934554d78a20e0288eafff12f (DEFAULT_TABLE table=test-table [id=33cd59491fd74f91b323764b8c931b57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:44.599888 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fc4ceb934554d78a20e0288eafff12f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:44.600778 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:44.594816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45009 (local address 127.0.0.1:43970)
0924 07:09:44.595282 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:44.595290 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:44.595336 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:44.595717 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:44.595717 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:44.595719 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:44.595759 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:44.595760 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.596794 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.596795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:44.598201 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:44.598203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.599116 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.599117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:44.599177 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:44.599181 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:09:44.599238 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 07:09:44.600619 (+ 1381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":7,"thread_start_us":64,"threads_started":1}
I20260924 07:09:44.600845 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 3fc4ceb934554d78a20e0288eafff12f (DEFAULT_TABLE table=test-table [id=33cd59491fd74f91b323764b8c931b57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:44.601897 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fc4ceb934554d78a20e0288eafff12f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:44.601918 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 3fc4ceb934554d78a20e0288eafff12f (DEFAULT_TABLE table=test-table [id=33cd59491fd74f91b323764b8c931b57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:44.602365 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fc4ceb934554d78a20e0288eafff12f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:44.603405 2628 tablet_bootstrap.cc:492] T 3fc4ceb934554d78a20e0288eafff12f P a5cc149836284c468c0bff3680e18479: Bootstrap starting.
I20260924 07:09:44.604002 2628 tablet_bootstrap.cc:654] T 3fc4ceb934554d78a20e0288eafff12f P a5cc149836284c468c0bff3680e18479: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:44.604297 2628 log.cc:824] T 3fc4ceb934554d78a20e0288eafff12f P a5cc149836284c468c0bff3680e18479: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:44.605078 2628 tablet_bootstrap.cc:492] T 3fc4ceb934554d78a20e0288eafff12f P a5cc149836284c468c0bff3680e18479: No bootstrap required, opened a new log
I20260924 07:09:44.605145 2628 ts_tablet_manager.cc:1397] T 3fc4ceb934554d78a20e0288eafff12f P a5cc149836284c468c0bff3680e18479: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:09:44.606003 2631 tablet_bootstrap.cc:492] T 3fc4ceb934554d78a20e0288eafff12f P ae93f97d901b4fa0aba5531618b11c0c: Bootstrap starting.
I20260924 07:09:44.606786 2631 tablet_bootstrap.cc:654] T 3fc4ceb934554d78a20e0288eafff12f P ae93f97d901b4fa0aba5531618b11c0c: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:44.606917 2628 raft_consensus.cc:348] T 3fc4ceb934554d78a20e0288eafff12f P a5cc149836284c468c0bff3680e18479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } }
I20260924 07:09:44.607095 2631 log.cc:824] T 3fc4ceb934554d78a20e0288eafff12f P ae93f97d901b4fa0aba5531618b11c0c: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:44.607110 2628 raft_consensus.cc:374] T 3fc4ceb934554d78a20e0288eafff12f P a5cc149836284c468c0bff3680e18479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:44.607144 2628 raft_consensus.cc:729] T 3fc4ceb934554d78a20e0288eafff12f P a5cc149836284c468c0bff3680e18479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5cc149836284c468c0bff3680e18479, State: Initialized, Role: FOLLOWER
I20260924 07:09:44.607259 2628 consensus_queue.cc:260] T 3fc4ceb934554d78a20e0288eafff12f P a5cc149836284c468c0bff3680e18479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } }
I20260924 07:09:44.607445 2630 tablet_bootstrap.cc:492] T 3fc4ceb934554d78a20e0288eafff12f P 9afadad6d73540b88766e90a1684b61f: Bootstrap starting.
I20260924 07:09:44.607534 2628 ts_tablet_manager.cc:1428] T 3fc4ceb934554d78a20e0288eafff12f P a5cc149836284c468c0bff3680e18479: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 07:09:44.607669 2605 heartbeater.cc:503] Master 127.0.98.62:37287 was elected leader, sending a full tablet report...
I20260924 07:09:44.607874 2631 tablet_bootstrap.cc:492] T 3fc4ceb934554d78a20e0288eafff12f P ae93f97d901b4fa0aba5531618b11c0c: No bootstrap required, opened a new log
I20260924 07:09:44.607957 2631 ts_tablet_manager.cc:1397] T 3fc4ceb934554d78a20e0288eafff12f P ae93f97d901b4fa0aba5531618b11c0c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:09:44.608094 2630 tablet_bootstrap.cc:654] T 3fc4ceb934554d78a20e0288eafff12f P 9afadad6d73540b88766e90a1684b61f: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:44.608454 2630 log.cc:824] T 3fc4ceb934554d78a20e0288eafff12f P 9afadad6d73540b88766e90a1684b61f: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:44.609156 2630 tablet_bootstrap.cc:492] T 3fc4ceb934554d78a20e0288eafff12f P 9afadad6d73540b88766e90a1684b61f: No bootstrap required, opened a new log
I20260924 07:09:44.609208 2630 ts_tablet_manager.cc:1397] T 3fc4ceb934554d78a20e0288eafff12f P 9afadad6d73540b88766e90a1684b61f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:09:44.609781 2631 raft_consensus.cc:348] T 3fc4ceb934554d78a20e0288eafff12f P ae93f97d901b4fa0aba5531618b11c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } }
I20260924 07:09:44.609931 2631 raft_consensus.cc:374] T 3fc4ceb934554d78a20e0288eafff12f P ae93f97d901b4fa0aba5531618b11c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:44.609974 2631 raft_consensus.cc:729] T 3fc4ceb934554d78a20e0288eafff12f P ae93f97d901b4fa0aba5531618b11c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae93f97d901b4fa0aba5531618b11c0c, State: Initialized, Role: FOLLOWER
I20260924 07:09:44.610085 2631 consensus_queue.cc:260] T 3fc4ceb934554d78a20e0288eafff12f P ae93f97d901b4fa0aba5531618b11c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } }
I20260924 07:09:44.610376 2631 ts_tablet_manager.cc:1428] T 3fc4ceb934554d78a20e0288eafff12f P ae93f97d901b4fa0aba5531618b11c0c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 07:09:44.610417 2325 heartbeater.cc:503] Master 127.0.98.62:37287 was elected leader, sending a full tablet report...
I20260924 07:09:44.610858 2630 raft_consensus.cc:348] T 3fc4ceb934554d78a20e0288eafff12f P 9afadad6d73540b88766e90a1684b61f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } }
I20260924 07:09:44.611030 2630 raft_consensus.cc:374] T 3fc4ceb934554d78a20e0288eafff12f P 9afadad6d73540b88766e90a1684b61f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:44.611064 2630 raft_consensus.cc:729] T 3fc4ceb934554d78a20e0288eafff12f P 9afadad6d73540b88766e90a1684b61f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9afadad6d73540b88766e90a1684b61f, State: Initialized, Role: FOLLOWER
I20260924 07:09:44.611176 2630 consensus_queue.cc:260] T 3fc4ceb934554d78a20e0288eafff12f P 9afadad6d73540b88766e90a1684b61f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } }
I20260924 07:09:44.611478 2630 ts_tablet_manager.cc:1428] T 3fc4ceb934554d78a20e0288eafff12f P 9afadad6d73540b88766e90a1684b61f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:09:44.611513 2465 heartbeater.cc:503] Master 127.0.98.62:37287 was elected leader, sending a full tablet report...
W20260924 07:09:44.635583 2466 tablet.cc:2369] T 3fc4ceb934554d78a20e0288eafff12f P 9afadad6d73540b88766e90a1684b61f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:09:44.730767 2326 tablet.cc:2369] T 3fc4ceb934554d78a20e0288eafff12f P ae93f97d901b4fa0aba5531618b11c0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:09:44.804896 2606 tablet.cc:2369] T 3fc4ceb934554d78a20e0288eafff12f P a5cc149836284c468c0bff3680e18479: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:09:44.807999 2633 raft_consensus.cc:482] T 3fc4ceb934554d78a20e0288eafff12f P a5cc149836284c468c0bff3680e18479 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:09:44.808091 2633 raft_consensus.cc:504] T 3fc4ceb934554d78a20e0288eafff12f P a5cc149836284c468c0bff3680e18479 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } }
I20260924 07:09:44.808537 2633 leader_election.cc:288] T 3fc4ceb934554d78a20e0288eafff12f P a5cc149836284c468c0bff3680e18479 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae93f97d901b4fa0aba5531618b11c0c (127.0.98.1:35995), 9afadad6d73540b88766e90a1684b61f (127.0.98.2:45009)
I20260924 07:09:44.811520 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:44.808808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36351 (local address 127.0.98.1:35995)
0924 07:09:44.808865 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:44.808867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:44.808878 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:09:44.808964 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:44.808966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:44.808979 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:44.809059 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:44.809225 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.809775 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.810693 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.811204 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.811255 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:44.811263 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:44.811441 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 07:09:44.811469 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 07:09:44.811538 2498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:44.808711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35995 (local address 127.0.98.3:36351)
0924 07:09:44.808787 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:44.808799 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:44.808884 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:44.809063 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:44.809064 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:44.809065 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:44.809154 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:44.809156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.809852 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.809854 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:44.810610 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:44.810612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.811271 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.811272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:44.811350 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:44.811354 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:09:44.811414 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 07:09:44.811449 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 07:09:44.811869 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc4ceb934554d78a20e0288eafff12f" candidate_uuid: "a5cc149836284c468c0bff3680e18479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae93f97d901b4fa0aba5531618b11c0c" is_pre_election: true
I20260924 07:09:44.811978 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:44.808945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44107 (local address 127.0.98.2:45009)
0924 07:09:44.808993 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:44.808995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:44.809105 (+ 110us) server_negotiation.cc:407] Connection header received
0924 07:09:44.809154 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:44.809155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:44.809167 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:44.809224 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:44.809354 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.809793 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.811106 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.811600 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.811670 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:44.811680 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:44.811885 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 07:09:44.811916 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 07:09:44.812052 2279 raft_consensus.cc:2454] T 3fc4ceb934554d78a20e0288eafff12f P ae93f97d901b4fa0aba5531618b11c0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5cc149836284c468c0bff3680e18479 in term 0.
I20260924 07:09:44.811991 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:44.808863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45009 (local address 127.0.98.3:44107)
0924 07:09:44.809036 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:44.809045 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:44.809101 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:44.809224 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:44.809224 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:44.809226 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:44.809297 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:44.809299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.809897 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.809898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:44.811009 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:44.811011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.811727 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.811727 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:44.811800 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:44.811803 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:09:44.811867 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 07:09:44.811904 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":113,"threads_started":1}
I20260924 07:09:44.812413 2493 leader_election.cc:302] T 3fc4ceb934554d78a20e0288eafff12f P a5cc149836284c468c0bff3680e18479 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5cc149836284c468c0bff3680e18479, ae93f97d901b4fa0aba5531618b11c0c; no voters:
I20260924 07:09:44.812963 2633 raft_consensus.cc:2781] T 3fc4ceb934554d78a20e0288eafff12f P a5cc149836284c468c0bff3680e18479 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:09:44.813017 2633 raft_consensus.cc:482] T 3fc4ceb934554d78a20e0288eafff12f P a5cc149836284c468c0bff3680e18479 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:09:44.813046 2633 raft_consensus.cc:3037] T 3fc4ceb934554d78a20e0288eafff12f P a5cc149836284c468c0bff3680e18479 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:44.813216 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc4ceb934554d78a20e0288eafff12f" candidate_uuid: "a5cc149836284c468c0bff3680e18479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9afadad6d73540b88766e90a1684b61f" is_pre_election: true
I20260924 07:09:44.813355 2419 raft_consensus.cc:2454] T 3fc4ceb934554d78a20e0288eafff12f P 9afadad6d73540b88766e90a1684b61f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5cc149836284c468c0bff3680e18479 in term 0.
I20260924 07:09:44.813901 2633 raft_consensus.cc:504] T 3fc4ceb934554d78a20e0288eafff12f P a5cc149836284c468c0bff3680e18479 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } }
I20260924 07:09:44.814029 2633 leader_election.cc:288] T 3fc4ceb934554d78a20e0288eafff12f P a5cc149836284c468c0bff3680e18479 [CANDIDATE]: Term 1 election: Requested vote from peers ae93f97d901b4fa0aba5531618b11c0c (127.0.98.1:35995), 9afadad6d73540b88766e90a1684b61f (127.0.98.2:45009)
I20260924 07:09:44.814170 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc4ceb934554d78a20e0288eafff12f" candidate_uuid: "a5cc149836284c468c0bff3680e18479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae93f97d901b4fa0aba5531618b11c0c"
I20260924 07:09:44.814188 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc4ceb934554d78a20e0288eafff12f" candidate_uuid: "a5cc149836284c468c0bff3680e18479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9afadad6d73540b88766e90a1684b61f"
I20260924 07:09:44.814245 2279 raft_consensus.cc:3037] T 3fc4ceb934554d78a20e0288eafff12f P ae93f97d901b4fa0aba5531618b11c0c [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:44.814273 2419 raft_consensus.cc:3037] T 3fc4ceb934554d78a20e0288eafff12f P 9afadad6d73540b88766e90a1684b61f [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:44.814929 2279 raft_consensus.cc:2454] T 3fc4ceb934554d78a20e0288eafff12f P ae93f97d901b4fa0aba5531618b11c0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5cc149836284c468c0bff3680e18479 in term 1.
I20260924 07:09:44.815062 2419 raft_consensus.cc:2454] T 3fc4ceb934554d78a20e0288eafff12f P 9afadad6d73540b88766e90a1684b61f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5cc149836284c468c0bff3680e18479 in term 1.
I20260924 07:09:44.815133 2493 leader_election.cc:302] T 3fc4ceb934554d78a20e0288eafff12f P a5cc149836284c468c0bff3680e18479 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5cc149836284c468c0bff3680e18479, ae93f97d901b4fa0aba5531618b11c0c; no voters:
I20260924 07:09:44.815271 2633 raft_consensus.cc:2781] T 3fc4ceb934554d78a20e0288eafff12f P a5cc149836284c468c0bff3680e18479 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:09:44.815465 2633 raft_consensus.cc:686] T 3fc4ceb934554d78a20e0288eafff12f P a5cc149836284c468c0bff3680e18479 [term 1 LEADER]: Becoming Leader. State: Replica: a5cc149836284c468c0bff3680e18479, State: Running, Role: LEADER
I20260924 07:09:44.815579 2633 consensus_queue.cc:237] T 3fc4ceb934554d78a20e0288eafff12f P a5cc149836284c468c0bff3680e18479 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } }
I20260924 07:09:44.816607 2122 catalog_manager.cc:5495] T 3fc4ceb934554d78a20e0288eafff12f P a5cc149836284c468c0bff3680e18479 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5cc149836284c468c0bff3680e18479 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a5cc149836284c468c0bff3680e18479" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } health_report { overall_health: UNKNOWN } } }
Sep 24 07:09:44 dist-test-slave-14ww krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:09:44.853397 2182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:44.845419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39742 (local address 127.0.98.62:37287)
0924 07:09:44.845493 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:44.845495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:44.845627 (+ 132us) server_negotiation.cc:407] Connection header received
0924 07:09:44.845668 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:44.845669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:44.845677 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:44.845728 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:44.846172 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.846674 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.847340 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.847527 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.847583 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:44.850251 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:44.850252 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:44.850253 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:44.850259 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:44.852403 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:44.852805 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:44.852806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:44.852806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:44.852854 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:44.853021 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:44.853021 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:44.853022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:44.853115 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:44.853162 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:44.853273 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 07:09:44.853335 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 07:09:44.853507 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:44.845374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37287 (local address 127.0.0.1:39742)
0924 07:09:44.845551 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:44.845561 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:44.845620 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:44.845741 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:44.845741 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:44.846032 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:44.846074 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:44.846076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.846746 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.846747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:44.847258 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:44.847259 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:44.847594 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:44.847595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:44.847631 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:44.848900 (+ 1269us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:44.848904 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:44.850173 (+ 1269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:44.852482 (+ 2309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:44.852489 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:44.852491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:44.852678 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:44.852913 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:44.852914 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:44.852914 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:44.852968 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:44.853166 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:44.853171 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:44.853209 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 07:09:44.853259 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 07:09:44.853383 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":46,"thread_start_us":111,"threads_started":1}
I20260924 07:09:44.855999 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39742:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 07:09:44.856222 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:09:44.857692 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 7471172ca22e4ca492fe6343044c4694 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a329f46a73f74631989e7cb7e963c8c7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:09:44.857695 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 7471172ca22e4ca492fe6343044c4694 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a329f46a73f74631989e7cb7e963c8c7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:09:44.857789 2539 tablet_service.cc:1467] Processing CreateTablet for tablet 7471172ca22e4ca492fe6343044c4694 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a329f46a73f74631989e7cb7e963c8c7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:09:44.857857 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7471172ca22e4ca492fe6343044c4694. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:44.857875 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7471172ca22e4ca492fe6343044c4694. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:44.857908 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7471172ca22e4ca492fe6343044c4694. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:44.859104 2630 tablet_bootstrap.cc:492] T 7471172ca22e4ca492fe6343044c4694 P 9afadad6d73540b88766e90a1684b61f: Bootstrap starting.
I20260924 07:09:44.859404 2628 tablet_bootstrap.cc:492] T 7471172ca22e4ca492fe6343044c4694 P a5cc149836284c468c0bff3680e18479: Bootstrap starting.
I20260924 07:09:44.859823 2628 tablet_bootstrap.cc:654] T 7471172ca22e4ca492fe6343044c4694 P a5cc149836284c468c0bff3680e18479: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:44.859843 2630 tablet_bootstrap.cc:654] T 7471172ca22e4ca492fe6343044c4694 P 9afadad6d73540b88766e90a1684b61f: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:44.860193 2631 tablet_bootstrap.cc:492] T 7471172ca22e4ca492fe6343044c4694 P ae93f97d901b4fa0aba5531618b11c0c: Bootstrap starting.
I20260924 07:09:44.860534 2631 tablet_bootstrap.cc:654] T 7471172ca22e4ca492fe6343044c4694 P ae93f97d901b4fa0aba5531618b11c0c: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:44.860538 2628 tablet_bootstrap.cc:492] T 7471172ca22e4ca492fe6343044c4694 P a5cc149836284c468c0bff3680e18479: No bootstrap required, opened a new log
I20260924 07:09:44.860600 2628 ts_tablet_manager.cc:1397] T 7471172ca22e4ca492fe6343044c4694 P a5cc149836284c468c0bff3680e18479: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:09:44.860819 2630 tablet_bootstrap.cc:492] T 7471172ca22e4ca492fe6343044c4694 P 9afadad6d73540b88766e90a1684b61f: No bootstrap required, opened a new log
I20260924 07:09:44.860816 2628 raft_consensus.cc:348] T 7471172ca22e4ca492fe6343044c4694 P a5cc149836284c468c0bff3680e18479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } }
I20260924 07:09:44.860863 2630 ts_tablet_manager.cc:1397] T 7471172ca22e4ca492fe6343044c4694 P 9afadad6d73540b88766e90a1684b61f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:09:44.860881 2628 raft_consensus.cc:374] T 7471172ca22e4ca492fe6343044c4694 P a5cc149836284c468c0bff3680e18479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:44.860896 2628 raft_consensus.cc:729] T 7471172ca22e4ca492fe6343044c4694 P a5cc149836284c468c0bff3680e18479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5cc149836284c468c0bff3680e18479, State: Initialized, Role: FOLLOWER
I20260924 07:09:44.860953 2628 consensus_queue.cc:260] T 7471172ca22e4ca492fe6343044c4694 P a5cc149836284c468c0bff3680e18479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } }
I20260924 07:09:44.861016 2630 raft_consensus.cc:348] T 7471172ca22e4ca492fe6343044c4694 P 9afadad6d73540b88766e90a1684b61f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } }
I20260924 07:09:44.861064 2628 ts_tablet_manager.cc:1428] T 7471172ca22e4ca492fe6343044c4694 P a5cc149836284c468c0bff3680e18479: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 07:09:44.861094 2630 raft_consensus.cc:374] T 7471172ca22e4ca492fe6343044c4694 P 9afadad6d73540b88766e90a1684b61f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:44.861120 2630 raft_consensus.cc:729] T 7471172ca22e4ca492fe6343044c4694 P 9afadad6d73540b88766e90a1684b61f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9afadad6d73540b88766e90a1684b61f, State: Initialized, Role: FOLLOWER
I20260924 07:09:44.861104 2633 tablet_replica.cc:439] T 7471172ca22e4ca492fe6343044c4694 P a5cc149836284c468c0bff3680e18479: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } } }
I20260924 07:09:44.861176 2633 tablet_replica.cc:442] T 7471172ca22e4ca492fe6343044c4694 P a5cc149836284c468c0bff3680e18479: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:09:44.861173 2630 consensus_queue.cc:260] T 7471172ca22e4ca492fe6343044c4694 P 9afadad6d73540b88766e90a1684b61f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } }
I20260924 07:09:44.861244 2631 tablet_bootstrap.cc:492] T 7471172ca22e4ca492fe6343044c4694 P ae93f97d901b4fa0aba5531618b11c0c: No bootstrap required, opened a new log
I20260924 07:09:44.861267 2630 ts_tablet_manager.cc:1428] T 7471172ca22e4ca492fe6343044c4694 P 9afadad6d73540b88766e90a1684b61f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:44.861282 2631 ts_tablet_manager.cc:1397] T 7471172ca22e4ca492fe6343044c4694 P ae93f97d901b4fa0aba5531618b11c0c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 07:09:44.861316 2636 tablet_replica.cc:439] T 7471172ca22e4ca492fe6343044c4694 P 9afadad6d73540b88766e90a1684b61f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } } }
I20260924 07:09:44.861433 2636 tablet_replica.cc:442] T 7471172ca22e4ca492fe6343044c4694 P 9afadad6d73540b88766e90a1684b61f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:09:44.861438 2631 raft_consensus.cc:348] T 7471172ca22e4ca492fe6343044c4694 P ae93f97d901b4fa0aba5531618b11c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } }
I20260924 07:09:44.861509 2631 raft_consensus.cc:374] T 7471172ca22e4ca492fe6343044c4694 P ae93f97d901b4fa0aba5531618b11c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:44.861536 2631 raft_consensus.cc:729] T 7471172ca22e4ca492fe6343044c4694 P ae93f97d901b4fa0aba5531618b11c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae93f97d901b4fa0aba5531618b11c0c, State: Initialized, Role: FOLLOWER
I20260924 07:09:44.861575 2631 consensus_queue.cc:260] T 7471172ca22e4ca492fe6343044c4694 P ae93f97d901b4fa0aba5531618b11c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } }
I20260924 07:09:44.861685 2631 ts_tablet_manager.cc:1428] T 7471172ca22e4ca492fe6343044c4694 P ae93f97d901b4fa0aba5531618b11c0c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:44.861791 2635 tablet_replica.cc:439] T 7471172ca22e4ca492fe6343044c4694 P ae93f97d901b4fa0aba5531618b11c0c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } } }
I20260924 07:09:44.861892 2635 tablet_replica.cc:442] T 7471172ca22e4ca492fe6343044c4694 P ae93f97d901b4fa0aba5531618b11c0c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:09:45.142932 2636 raft_consensus.cc:482] T 7471172ca22e4ca492fe6343044c4694 P 9afadad6d73540b88766e90a1684b61f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:09:45.143069 2636 raft_consensus.cc:504] T 7471172ca22e4ca492fe6343044c4694 P 9afadad6d73540b88766e90a1684b61f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } }
I20260924 07:09:45.143399 2636 leader_election.cc:288] T 7471172ca22e4ca492fe6343044c4694 P 9afadad6d73540b88766e90a1684b61f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae93f97d901b4fa0aba5531618b11c0c (127.0.98.1:35995), a5cc149836284c468c0bff3680e18479 (127.0.98.3:37165)
I20260924 07:09:45.146852 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:45.143669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35209 (local address 127.0.98.1:35995)
0924 07:09:45.143730 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:45.143733 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:45.143817 (+ 84us) server_negotiation.cc:407] Connection header received
0924 07:09:45.143880 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:45.143881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:45.143892 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:45.143958 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:45.144245 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:45.145047 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:45.146129 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:45.146495 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:45.146579 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:45.146589 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:45.146736 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 07:09:45.146777 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 07:09:45.146899 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:45.143579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35995 (local address 127.0.98.2:35209)
0924 07:09:45.143733 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:45.143746 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:45.143822 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:45.144041 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:45.144042 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:45.144046 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:45.144156 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:45.144159 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:45.145178 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:45.145179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:45.146032 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:45.146034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:45.146579 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:45.146580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:45.146646 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:45.146650 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:09:45.146728 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 07:09:45.146770 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":43,"thread_start_us":59,"threads_started":1}
I20260924 07:09:45.146919 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:45.143579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37165 (local address 127.0.98.2:55911)
0924 07:09:45.143750 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:45.143769 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:45.143818 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:45.144041 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:45.144042 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:45.144045 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:45.144156 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:45.144159 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:45.145203 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:45.145204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:45.146056 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:45.146058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:45.146621 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:45.146622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:45.146693 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:45.146697 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:09:45.146772 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 07:09:45.146829 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":75,"thread_start_us":43,"threads_started":1}
I20260924 07:09:45.146910 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:45.143652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55911 (local address 127.0.98.3:37165)
0924 07:09:45.143759 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:45.143761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:45.143814 (+ 53us) server_negotiation.cc:407] Connection header received
0924 07:09:45.143902 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:45.143903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:45.143914 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:45.143982 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:45.144268 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:45.145056 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:45.146147 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:45.146535 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:45.146619 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:45.146630 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:45.146780 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 07:09:45.146829 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":56,"threads_started":1}
I20260924 07:09:45.147272 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7471172ca22e4ca492fe6343044c4694" candidate_uuid: "9afadad6d73540b88766e90a1684b61f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae93f97d901b4fa0aba5531618b11c0c" is_pre_election: true
I20260924 07:09:45.147362 2279 raft_consensus.cc:2454] T 7471172ca22e4ca492fe6343044c4694 P ae93f97d901b4fa0aba5531618b11c0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9afadad6d73540b88766e90a1684b61f in term 0.
I20260924 07:09:45.147598 2353 leader_election.cc:302] T 7471172ca22e4ca492fe6343044c4694 P 9afadad6d73540b88766e90a1684b61f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9afadad6d73540b88766e90a1684b61f, ae93f97d901b4fa0aba5531618b11c0c; no voters:
I20260924 07:09:45.147746 2636 raft_consensus.cc:2781] T 7471172ca22e4ca492fe6343044c4694 P 9afadad6d73540b88766e90a1684b61f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:09:45.147789 2636 raft_consensus.cc:482] T 7471172ca22e4ca492fe6343044c4694 P 9afadad6d73540b88766e90a1684b61f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:09:45.147729 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7471172ca22e4ca492fe6343044c4694" candidate_uuid: "9afadad6d73540b88766e90a1684b61f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5cc149836284c468c0bff3680e18479" is_pre_election: true
I20260924 07:09:45.147807 2636 raft_consensus.cc:3037] T 7471172ca22e4ca492fe6343044c4694 P 9afadad6d73540b88766e90a1684b61f [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:45.147848 2559 raft_consensus.cc:2454] T 7471172ca22e4ca492fe6343044c4694 P a5cc149836284c468c0bff3680e18479 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9afadad6d73540b88766e90a1684b61f in term 0.
I20260924 07:09:45.148595 2636 raft_consensus.cc:504] T 7471172ca22e4ca492fe6343044c4694 P 9afadad6d73540b88766e90a1684b61f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } }
I20260924 07:09:45.148741 2636 leader_election.cc:288] T 7471172ca22e4ca492fe6343044c4694 P 9afadad6d73540b88766e90a1684b61f [CANDIDATE]: Term 1 election: Requested vote from peers ae93f97d901b4fa0aba5531618b11c0c (127.0.98.1:35995), a5cc149836284c468c0bff3680e18479 (127.0.98.3:37165)
I20260924 07:09:45.148901 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7471172ca22e4ca492fe6343044c4694" candidate_uuid: "9afadad6d73540b88766e90a1684b61f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae93f97d901b4fa0aba5531618b11c0c"
I20260924 07:09:45.148969 2279 raft_consensus.cc:3037] T 7471172ca22e4ca492fe6343044c4694 P ae93f97d901b4fa0aba5531618b11c0c [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:45.148941 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7471172ca22e4ca492fe6343044c4694" candidate_uuid: "9afadad6d73540b88766e90a1684b61f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5cc149836284c468c0bff3680e18479"
I20260924 07:09:45.149019 2559 raft_consensus.cc:3037] T 7471172ca22e4ca492fe6343044c4694 P a5cc149836284c468c0bff3680e18479 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:45.149580 2279 raft_consensus.cc:2454] T 7471172ca22e4ca492fe6343044c4694 P ae93f97d901b4fa0aba5531618b11c0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9afadad6d73540b88766e90a1684b61f in term 1.
I20260924 07:09:45.149744 2559 raft_consensus.cc:2454] T 7471172ca22e4ca492fe6343044c4694 P a5cc149836284c468c0bff3680e18479 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9afadad6d73540b88766e90a1684b61f in term 1.
I20260924 07:09:45.149775 2353 leader_election.cc:302] T 7471172ca22e4ca492fe6343044c4694 P 9afadad6d73540b88766e90a1684b61f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9afadad6d73540b88766e90a1684b61f, ae93f97d901b4fa0aba5531618b11c0c; no voters:
I20260924 07:09:45.149887 2636 raft_consensus.cc:2781] T 7471172ca22e4ca492fe6343044c4694 P 9afadad6d73540b88766e90a1684b61f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:09:45.150002 2636 raft_consensus.cc:686] T 7471172ca22e4ca492fe6343044c4694 P 9afadad6d73540b88766e90a1684b61f [term 1 LEADER]: Becoming Leader. State: Replica: 9afadad6d73540b88766e90a1684b61f, State: Running, Role: LEADER
I20260924 07:09:45.150095 2636 consensus_queue.cc:237] T 7471172ca22e4ca492fe6343044c4694 P 9afadad6d73540b88766e90a1684b61f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } }
I20260924 07:09:45.150501 2652 tablet_replica.cc:439] T 7471172ca22e4ca492fe6343044c4694 P 9afadad6d73540b88766e90a1684b61f: TxnStatusTablet state changed. Reason: New leader 9afadad6d73540b88766e90a1684b61f. Latest consensus state: current_term: 1 leader_uuid: "9afadad6d73540b88766e90a1684b61f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } } }
I20260924 07:09:45.150594 2652 tablet_replica.cc:442] T 7471172ca22e4ca492fe6343044c4694 P 9afadad6d73540b88766e90a1684b61f: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:09:45.150815 2654 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 07:09:45.150934 2654 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 07:09:45.150890 2120 catalog_manager.cc:5495] T 7471172ca22e4ca492fe6343044c4694 P 9afadad6d73540b88766e90a1684b61f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9afadad6d73540b88766e90a1684b61f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "9afadad6d73540b88766e90a1684b61f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } health_report { overall_health: UNKNOWN } } }
I20260924 07:09:45.215191 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:45.212614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43978 (local address 127.0.98.2:45009)
0924 07:09:45.212690 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:45.212692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:45.212702 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:09:45.212829 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:45.212830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:45.212847 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:45.212939 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:09:45.213093 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:45.213602 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:45.214360 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:45.214573 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:45.214659 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:45.214841 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:09:45.214941 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:09:45.215029 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:45.215067 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 07:09:45.215088 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 07:09:45.215185 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:45.212540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45009 (local address 127.0.0.1:43978)
0924 07:09:45.212628 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:45.212658 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:45.212718 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:45.212945 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:45.212946 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:45.212947 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:09:45.213005 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:45.213007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:45.213688 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:45.213689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:45.214271 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:45.214273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:45.214660 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:45.214661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:45.214728 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:45.214748 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:09:45.215008 (+ 260us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:09:45.215016 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 07:09:45.215061 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 07:09:45.215112 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 07:09:45.217563 2279 raft_consensus.cc:1261] T 7471172ca22e4ca492fe6343044c4694 P ae93f97d901b4fa0aba5531618b11c0c [term 1 FOLLOWER]: Refusing update from remote peer 9afadad6d73540b88766e90a1684b61f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:09:45.218189 2652 consensus_queue.cc:1035] T 7471172ca22e4ca492fe6343044c4694 P 9afadad6d73540b88766e90a1684b61f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:09:45.217866 2559 raft_consensus.cc:1261] T 7471172ca22e4ca492fe6343044c4694 P a5cc149836284c468c0bff3680e18479 [term 1 FOLLOWER]: Refusing update from remote peer 9afadad6d73540b88766e90a1684b61f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:09:45.219014 2652 consensus_queue.cc:1035] T 7471172ca22e4ca492fe6343044c4694 P 9afadad6d73540b88766e90a1684b61f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:09:45.219617 2635 tablet_replica.cc:439] T 7471172ca22e4ca492fe6343044c4694 P ae93f97d901b4fa0aba5531618b11c0c: TxnStatusTablet state changed. Reason: New leader 9afadad6d73540b88766e90a1684b61f. Latest consensus state: current_term: 1 leader_uuid: "9afadad6d73540b88766e90a1684b61f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } } }
I20260924 07:09:45.219727 2635 tablet_replica.cc:442] T 7471172ca22e4ca492fe6343044c4694 P ae93f97d901b4fa0aba5531618b11c0c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:09:45.220042 2633 tablet_replica.cc:439] T 7471172ca22e4ca492fe6343044c4694 P a5cc149836284c468c0bff3680e18479: TxnStatusTablet state changed. Reason: New leader 9afadad6d73540b88766e90a1684b61f. Latest consensus state: current_term: 1 leader_uuid: "9afadad6d73540b88766e90a1684b61f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } } }
I20260924 07:09:45.220139 2633 tablet_replica.cc:442] T 7471172ca22e4ca492fe6343044c4694 P a5cc149836284c468c0bff3680e18479: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:09:45.220877 2636 tablet_replica.cc:439] T 7471172ca22e4ca492fe6343044c4694 P 9afadad6d73540b88766e90a1684b61f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9afadad6d73540b88766e90a1684b61f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } } }
I20260924 07:09:45.220997 2636 tablet_replica.cc:442] T 7471172ca22e4ca492fe6343044c4694 P 9afadad6d73540b88766e90a1684b61f: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:09:45.221294 2635 tablet_replica.cc:439] T 7471172ca22e4ca492fe6343044c4694 P ae93f97d901b4fa0aba5531618b11c0c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9afadad6d73540b88766e90a1684b61f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } } }
I20260924 07:09:45.221382 2635 tablet_replica.cc:442] T 7471172ca22e4ca492fe6343044c4694 P ae93f97d901b4fa0aba5531618b11c0c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:09:45.221474 2633 tablet_replica.cc:439] T 7471172ca22e4ca492fe6343044c4694 P a5cc149836284c468c0bff3680e18479: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9afadad6d73540b88766e90a1684b61f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } } }
I20260924 07:09:45.221555 2633 tablet_replica.cc:442] T 7471172ca22e4ca492fe6343044c4694 P a5cc149836284c468c0bff3680e18479: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:09:45.221859 2657 mvcc.cc:205] Tried to move back new op lower bound from 7332797584247513088 to 7332797583975280640. Current Snapshot: MvccSnapshot[applied={T|T < 7332797584247513088 or (T in {7332797584247513088})}]
I20260924 07:09:45.220875 2652 tablet_replica.cc:439] T 7471172ca22e4ca492fe6343044c4694 P 9afadad6d73540b88766e90a1684b61f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9afadad6d73540b88766e90a1684b61f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } } }
I20260924 07:09:45.222084 2652 tablet_replica.cc:442] T 7471172ca22e4ca492fe6343044c4694 P 9afadad6d73540b88766e90a1684b61f: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:09:45.222553 2658 mvcc.cc:205] Tried to move back new op lower bound from 7332797584247513088 to 7332797583975280640. Current Snapshot: MvccSnapshot[applied={T|T < 7332797584247513088}]
I20260924 07:09:45.223416 2633 consensus_queue.cc:1035] T 3fc4ceb934554d78a20e0288eafff12f P a5cc149836284c468c0bff3680e18479 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:09:45.224777 2638 consensus_queue.cc:1035] T 3fc4ceb934554d78a20e0288eafff12f P a5cc149836284c468c0bff3680e18479 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:09:45.226537 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:45.223249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53370 (local address 127.0.98.3:37165)
0924 07:09:45.223283 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:45.223285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:45.223330 (+ 45us) server_negotiation.cc:407] Connection header received
0924 07:09:45.223439 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:45.223440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:45.223457 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:45.223553 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:09:45.223758 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:45.224469 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:45.225629 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:45.225907 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:45.225984 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:45.226176 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:09:45.226276 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:09:45.226366 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:45.226433 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 07:09:45.226454 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 07:09:45.231935 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:45.228850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36685 (local address 127.0.98.2:45009)
0924 07:09:45.228894 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:45.228896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:45.229034 (+ 138us) server_negotiation.cc:407] Connection header received
0924 07:09:45.229060 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:45.229061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:45.229072 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:45.229140 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:09:45.229225 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:45.229913 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:45.231072 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:45.231372 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:45.231442 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:45.231660 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:09:45.231699 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:09:45.231760 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:45.231844 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 07:09:45.231863 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 07:09:45.231957 2673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:45.228805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45009 (local address 127.0.98.3:36685)
0924 07:09:45.228955 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:45.228965 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:45.229023 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:45.229140 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:45.229141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:45.229142 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:09:45.229189 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:45.229189 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:45.230005 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:45.230006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:45.230974 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:45.230977 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:45.231478 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:45.231479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:45.231543 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:45.231556 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:09:45.231776 (+ 220us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:09:45.231778 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 07:09:45.231829 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 07:09:45.231874 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260924 07:09:45.241428 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:45.238912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39593 (local address 127.0.98.3:37165)
0924 07:09:45.238949 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:45.238951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:45.238960 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:09:45.239029 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:45.239029 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:45.239037 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:45.239069 (+ 32us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:09:45.239203 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:45.239663 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:45.240761 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:45.240980 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:45.241029 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:45.241215 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:09:45.241237 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:09:45.241268 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:45.241365 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 07:09:45.241380 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 07:09:45.241503 2676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:45.238750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37165 (local address 127.0.98.2:39593)
0924 07:09:45.238905 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:45.238917 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:45.238966 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:45.239082 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:45.239082 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:45.239084 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:09:45.239147 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:45.239149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:45.239744 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:45.239745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:45.240660 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:45.240663 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:45.241053 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:45.241054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:45.241123 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:45.241136 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:09:45.241272 (+ 136us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:09:45.241273 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 07:09:45.241365 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 07:09:45.241408 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
I20260924 07:09:45.250473 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:09:45.250578 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:09:45.251267 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5283f28bbbbf47789cdb632a6923f1ba: Sending DeleteTablet for 3 replicas of tablet 3fc4ceb934554d78a20e0288eafff12f
I20260924 07:09:45.251545 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fc4ceb934554d78a20e0288eafff12f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53366
I20260924 07:09:45.251695 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fc4ceb934554d78a20e0288eafff12f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40000
I20260924 07:09:45.251868 2679 tablet_replica.cc:330] T 3fc4ceb934554d78a20e0288eafff12f P a5cc149836284c468c0bff3680e18479: stopping tablet replica
I20260924 07:09:45.251899 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fc4ceb934554d78a20e0288eafff12f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43970
I20260924 07:09:45.252043 2679 raft_consensus.cc:2229] T 3fc4ceb934554d78a20e0288eafff12f P a5cc149836284c468c0bff3680e18479 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:09:45.252173 2679 raft_consensus.cc:2258] T 3fc4ceb934554d78a20e0288eafff12f P a5cc149836284c468c0bff3680e18479 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:09:45.252324 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39732:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:09:45.252475 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:09:45.252821 2679 ts_tablet_manager.cc:1905] T 3fc4ceb934554d78a20e0288eafff12f P a5cc149836284c468c0bff3680e18479: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:09:45.253620 2539 tablet_service.cc:1467] Processing CreateTablet for tablet a174dfc5f00e46f7b41c7327f5739916 (DEFAULT_TABLE table=test-table [id=efd0fe2852184a88a20f8c71927dc961]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:45.253741 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a174dfc5f00e46f7b41c7327f5739916. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:45.254988 2679 ts_tablet_manager.cc:1918] T 3fc4ceb934554d78a20e0288eafff12f P a5cc149836284c468c0bff3680e18479: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:09:45.255018 2628 tablet_bootstrap.cc:492] T a174dfc5f00e46f7b41c7327f5739916 P a5cc149836284c468c0bff3680e18479: Bootstrap starting.
I20260924 07:09:45.255448 2399 tablet_service.cc:1467] Processing CreateTablet for tablet a174dfc5f00e46f7b41c7327f5739916 (DEFAULT_TABLE table=test-table [id=efd0fe2852184a88a20f8c71927dc961]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:45.255479 2628 tablet_bootstrap.cc:654] T a174dfc5f00e46f7b41c7327f5739916 P a5cc149836284c468c0bff3680e18479: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:45.255550 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a174dfc5f00e46f7b41c7327f5739916. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:45.255857 2679 log.cc:1196] T 3fc4ceb934554d78a20e0288eafff12f P a5cc149836284c468c0bff3680e18479: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-2/wal/wals/3fc4ceb934554d78a20e0288eafff12f
I20260924 07:09:45.256172 2679 ts_tablet_manager.cc:1939] T 3fc4ceb934554d78a20e0288eafff12f P a5cc149836284c468c0bff3680e18479: Deleting consensus metadata
I20260924 07:09:45.256596 2107 catalog_manager.cc:4841] TS a5cc149836284c468c0bff3680e18479 (127.0.98.3:37165): tablet 3fc4ceb934554d78a20e0288eafff12f (table test-table [id=33cd59491fd74f91b323764b8c931b57]) successfully deleted
I20260924 07:09:45.256904 2630 tablet_bootstrap.cc:492] T a174dfc5f00e46f7b41c7327f5739916 P 9afadad6d73540b88766e90a1684b61f: Bootstrap starting.
I20260924 07:09:45.257066 2628 tablet_bootstrap.cc:492] T a174dfc5f00e46f7b41c7327f5739916 P a5cc149836284c468c0bff3680e18479: No bootstrap required, opened a new log
I20260924 07:09:45.257107 2628 ts_tablet_manager.cc:1397] T a174dfc5f00e46f7b41c7327f5739916 P a5cc149836284c468c0bff3680e18479: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:09:45.257234 2628 raft_consensus.cc:348] T a174dfc5f00e46f7b41c7327f5739916 P a5cc149836284c468c0bff3680e18479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } }
I20260924 07:09:45.257292 2628 raft_consensus.cc:374] T a174dfc5f00e46f7b41c7327f5739916 P a5cc149836284c468c0bff3680e18479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:45.257315 2628 raft_consensus.cc:729] T a174dfc5f00e46f7b41c7327f5739916 P a5cc149836284c468c0bff3680e18479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5cc149836284c468c0bff3680e18479, State: Initialized, Role: FOLLOWER
I20260924 07:09:45.257324 2630 tablet_bootstrap.cc:654] T a174dfc5f00e46f7b41c7327f5739916 P 9afadad6d73540b88766e90a1684b61f: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:45.257364 2628 consensus_queue.cc:260] T a174dfc5f00e46f7b41c7327f5739916 P a5cc149836284c468c0bff3680e18479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } }
I20260924 07:09:45.257462 2628 ts_tablet_manager.cc:1428] T a174dfc5f00e46f7b41c7327f5739916 P a5cc149836284c468c0bff3680e18479: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:45.257769 2259 tablet_service.cc:1467] Processing CreateTablet for tablet a174dfc5f00e46f7b41c7327f5739916 (DEFAULT_TABLE table=test-table [id=efd0fe2852184a88a20f8c71927dc961]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:45.257895 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a174dfc5f00e46f7b41c7327f5739916. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:45.259258 2631 tablet_bootstrap.cc:492] T a174dfc5f00e46f7b41c7327f5739916 P ae93f97d901b4fa0aba5531618b11c0c: Bootstrap starting.
I20260924 07:09:45.259639 2631 tablet_bootstrap.cc:654] T a174dfc5f00e46f7b41c7327f5739916 P ae93f97d901b4fa0aba5531618b11c0c: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:45.260368 2631 tablet_bootstrap.cc:492] T a174dfc5f00e46f7b41c7327f5739916 P ae93f97d901b4fa0aba5531618b11c0c: No bootstrap required, opened a new log
I20260924 07:09:45.260428 2631 ts_tablet_manager.cc:1397] T a174dfc5f00e46f7b41c7327f5739916 P ae93f97d901b4fa0aba5531618b11c0c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 07:09:45.260463 2681 tablet_replica.cc:330] T 3fc4ceb934554d78a20e0288eafff12f P 9afadad6d73540b88766e90a1684b61f: stopping tablet replica
I20260924 07:09:45.260571 2681 raft_consensus.cc:2229] T 3fc4ceb934554d78a20e0288eafff12f P 9afadad6d73540b88766e90a1684b61f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:09:45.260614 2681 raft_consensus.cc:2258] T 3fc4ceb934554d78a20e0288eafff12f P 9afadad6d73540b88766e90a1684b61f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:09:45.260572 2631 raft_consensus.cc:348] T a174dfc5f00e46f7b41c7327f5739916 P ae93f97d901b4fa0aba5531618b11c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } }
I20260924 07:09:45.260644 2631 raft_consensus.cc:374] T a174dfc5f00e46f7b41c7327f5739916 P ae93f97d901b4fa0aba5531618b11c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:45.260665 2631 raft_consensus.cc:729] T a174dfc5f00e46f7b41c7327f5739916 P ae93f97d901b4fa0aba5531618b11c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae93f97d901b4fa0aba5531618b11c0c, State: Initialized, Role: FOLLOWER
I20260924 07:09:45.260708 2631 consensus_queue.cc:260] T a174dfc5f00e46f7b41c7327f5739916 P ae93f97d901b4fa0aba5531618b11c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } }
I20260924 07:09:45.260812 2631 ts_tablet_manager.cc:1428] T a174dfc5f00e46f7b41c7327f5739916 P ae93f97d901b4fa0aba5531618b11c0c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:45.260833 2680 tablet_replica.cc:330] T 3fc4ceb934554d78a20e0288eafff12f P ae93f97d901b4fa0aba5531618b11c0c: stopping tablet replica
I20260924 07:09:45.260931 2680 raft_consensus.cc:2229] T 3fc4ceb934554d78a20e0288eafff12f P ae93f97d901b4fa0aba5531618b11c0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:09:45.260993 2680 raft_consensus.cc:2258] T 3fc4ceb934554d78a20e0288eafff12f P ae93f97d901b4fa0aba5531618b11c0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:09:45.261484 2680 ts_tablet_manager.cc:1905] T 3fc4ceb934554d78a20e0288eafff12f P ae93f97d901b4fa0aba5531618b11c0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:09:45.261473 2630 tablet_bootstrap.cc:492] T a174dfc5f00e46f7b41c7327f5739916 P 9afadad6d73540b88766e90a1684b61f: No bootstrap required, opened a new log
I20260924 07:09:45.261543 2630 ts_tablet_manager.cc:1397] T a174dfc5f00e46f7b41c7327f5739916 P 9afadad6d73540b88766e90a1684b61f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260924 07:09:45.261687 2630 raft_consensus.cc:348] T a174dfc5f00e46f7b41c7327f5739916 P 9afadad6d73540b88766e90a1684b61f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } }
I20260924 07:09:45.261761 2630 raft_consensus.cc:374] T a174dfc5f00e46f7b41c7327f5739916 P 9afadad6d73540b88766e90a1684b61f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:45.261782 2630 raft_consensus.cc:729] T a174dfc5f00e46f7b41c7327f5739916 P 9afadad6d73540b88766e90a1684b61f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9afadad6d73540b88766e90a1684b61f, State: Initialized, Role: FOLLOWER
I20260924 07:09:45.261824 2630 consensus_queue.cc:260] T a174dfc5f00e46f7b41c7327f5739916 P 9afadad6d73540b88766e90a1684b61f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } }
I20260924 07:09:45.261914 2630 ts_tablet_manager.cc:1428] T a174dfc5f00e46f7b41c7327f5739916 P 9afadad6d73540b88766e90a1684b61f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:45.261487 2681 ts_tablet_manager.cc:1905] T 3fc4ceb934554d78a20e0288eafff12f P 9afadad6d73540b88766e90a1684b61f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:09:45.263136 2680 ts_tablet_manager.cc:1918] T 3fc4ceb934554d78a20e0288eafff12f P ae93f97d901b4fa0aba5531618b11c0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:09:45.263250 2680 log.cc:1196] T 3fc4ceb934554d78a20e0288eafff12f P ae93f97d901b4fa0aba5531618b11c0c: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-0/wal/wals/3fc4ceb934554d78a20e0288eafff12f
I20260924 07:09:45.263562 2680 ts_tablet_manager.cc:1939] T 3fc4ceb934554d78a20e0288eafff12f P ae93f97d901b4fa0aba5531618b11c0c: Deleting consensus metadata
I20260924 07:09:45.263751 2681 ts_tablet_manager.cc:1918] T 3fc4ceb934554d78a20e0288eafff12f P 9afadad6d73540b88766e90a1684b61f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:09:45.263820 2681 log.cc:1196] T 3fc4ceb934554d78a20e0288eafff12f P 9afadad6d73540b88766e90a1684b61f: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-1/wal/wals/3fc4ceb934554d78a20e0288eafff12f
I20260924 07:09:45.264075 2681 ts_tablet_manager.cc:1939] T 3fc4ceb934554d78a20e0288eafff12f P 9afadad6d73540b88766e90a1684b61f: Deleting consensus metadata
I20260924 07:09:45.264065 2109 catalog_manager.cc:4841] TS ae93f97d901b4fa0aba5531618b11c0c (127.0.98.1:35995): tablet 3fc4ceb934554d78a20e0288eafff12f (table test-table [id=33cd59491fd74f91b323764b8c931b57]) successfully deleted
I20260924 07:09:45.264549 2109 catalog_manager.cc:4841] TS 9afadad6d73540b88766e90a1684b61f (127.0.98.2:45009): tablet 3fc4ceb934554d78a20e0288eafff12f (table test-table [id=33cd59491fd74f91b323764b8c931b57]) successfully deleted
I20260924 07:09:45.298418 2633 raft_consensus.cc:482] T a174dfc5f00e46f7b41c7327f5739916 P a5cc149836284c468c0bff3680e18479 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:09:45.298521 2633 raft_consensus.cc:504] T a174dfc5f00e46f7b41c7327f5739916 P a5cc149836284c468c0bff3680e18479 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } }
I20260924 07:09:45.298691 2633 leader_election.cc:288] T a174dfc5f00e46f7b41c7327f5739916 P a5cc149836284c468c0bff3680e18479 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae93f97d901b4fa0aba5531618b11c0c (127.0.98.1:35995), 9afadad6d73540b88766e90a1684b61f (127.0.98.2:45009)
I20260924 07:09:45.298939 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a174dfc5f00e46f7b41c7327f5739916" candidate_uuid: "a5cc149836284c468c0bff3680e18479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9afadad6d73540b88766e90a1684b61f" is_pre_election: true
I20260924 07:09:45.298926 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a174dfc5f00e46f7b41c7327f5739916" candidate_uuid: "a5cc149836284c468c0bff3680e18479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae93f97d901b4fa0aba5531618b11c0c" is_pre_election: true
I20260924 07:09:45.299021 2419 raft_consensus.cc:2454] T a174dfc5f00e46f7b41c7327f5739916 P 9afadad6d73540b88766e90a1684b61f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5cc149836284c468c0bff3680e18479 in term 0.
I20260924 07:09:45.299022 2279 raft_consensus.cc:2454] T a174dfc5f00e46f7b41c7327f5739916 P ae93f97d901b4fa0aba5531618b11c0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5cc149836284c468c0bff3680e18479 in term 0.
I20260924 07:09:45.299199 2493 leader_election.cc:302] T a174dfc5f00e46f7b41c7327f5739916 P a5cc149836284c468c0bff3680e18479 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5cc149836284c468c0bff3680e18479, ae93f97d901b4fa0aba5531618b11c0c; no voters:
I20260924 07:09:45.299314 2633 raft_consensus.cc:2781] T a174dfc5f00e46f7b41c7327f5739916 P a5cc149836284c468c0bff3680e18479 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:09:45.299348 2633 raft_consensus.cc:482] T a174dfc5f00e46f7b41c7327f5739916 P a5cc149836284c468c0bff3680e18479 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:09:45.299372 2633 raft_consensus.cc:3037] T a174dfc5f00e46f7b41c7327f5739916 P a5cc149836284c468c0bff3680e18479 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:45.300215 2633 raft_consensus.cc:504] T a174dfc5f00e46f7b41c7327f5739916 P a5cc149836284c468c0bff3680e18479 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } }
I20260924 07:09:45.300334 2633 leader_election.cc:288] T a174dfc5f00e46f7b41c7327f5739916 P a5cc149836284c468c0bff3680e18479 [CANDIDATE]: Term 1 election: Requested vote from peers ae93f97d901b4fa0aba5531618b11c0c (127.0.98.1:35995), 9afadad6d73540b88766e90a1684b61f (127.0.98.2:45009)
I20260924 07:09:45.300537 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a174dfc5f00e46f7b41c7327f5739916" candidate_uuid: "a5cc149836284c468c0bff3680e18479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9afadad6d73540b88766e90a1684b61f"
I20260924 07:09:45.300534 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a174dfc5f00e46f7b41c7327f5739916" candidate_uuid: "a5cc149836284c468c0bff3680e18479" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae93f97d901b4fa0aba5531618b11c0c"
I20260924 07:09:45.300606 2279 raft_consensus.cc:3037] T a174dfc5f00e46f7b41c7327f5739916 P ae93f97d901b4fa0aba5531618b11c0c [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:45.300606 2419 raft_consensus.cc:3037] T a174dfc5f00e46f7b41c7327f5739916 P 9afadad6d73540b88766e90a1684b61f [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:45.301337 2419 raft_consensus.cc:2454] T a174dfc5f00e46f7b41c7327f5739916 P 9afadad6d73540b88766e90a1684b61f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5cc149836284c468c0bff3680e18479 in term 1.
I20260924 07:09:45.301367 2279 raft_consensus.cc:2454] T a174dfc5f00e46f7b41c7327f5739916 P ae93f97d901b4fa0aba5531618b11c0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5cc149836284c468c0bff3680e18479 in term 1.
I20260924 07:09:45.301497 2493 leader_election.cc:302] T a174dfc5f00e46f7b41c7327f5739916 P a5cc149836284c468c0bff3680e18479 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9afadad6d73540b88766e90a1684b61f, a5cc149836284c468c0bff3680e18479; no voters:
I20260924 07:09:45.301609 2633 raft_consensus.cc:2781] T a174dfc5f00e46f7b41c7327f5739916 P a5cc149836284c468c0bff3680e18479 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:09:45.301654 2633 raft_consensus.cc:686] T a174dfc5f00e46f7b41c7327f5739916 P a5cc149836284c468c0bff3680e18479 [term 1 LEADER]: Becoming Leader. State: Replica: a5cc149836284c468c0bff3680e18479, State: Running, Role: LEADER
I20260924 07:09:45.301702 2633 consensus_queue.cc:237] T a174dfc5f00e46f7b41c7327f5739916 P a5cc149836284c468c0bff3680e18479 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } }
I20260924 07:09:45.302254 2120 catalog_manager.cc:5495] T a174dfc5f00e46f7b41c7327f5739916 P a5cc149836284c468c0bff3680e18479 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5cc149836284c468c0bff3680e18479 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a5cc149836284c468c0bff3680e18479" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } health_report { overall_health: HEALTHY } } }
I20260924 07:09:45.309181 2419 raft_consensus.cc:1261] T a174dfc5f00e46f7b41c7327f5739916 P 9afadad6d73540b88766e90a1684b61f [term 1 FOLLOWER]: Refusing update from remote peer a5cc149836284c468c0bff3680e18479: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:09:45.309181 2279 raft_consensus.cc:1261] T a174dfc5f00e46f7b41c7327f5739916 P ae93f97d901b4fa0aba5531618b11c0c [term 1 FOLLOWER]: Refusing update from remote peer a5cc149836284c468c0bff3680e18479: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:09:45.309352 2633 consensus_queue.cc:1035] T a174dfc5f00e46f7b41c7327f5739916 P a5cc149836284c468c0bff3680e18479 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:09:45.309490 2638 consensus_queue.cc:1035] T a174dfc5f00e46f7b41c7327f5739916 P a5cc149836284c468c0bff3680e18479 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:09:45.310524 2660 mvcc.cc:205] Tried to move back new op lower bound from 7332797584625221632 to 7332797584596111360. Current Snapshot: MvccSnapshot[applied={T|T < 7332797584625221632}]
W20260924 07:09:45.313953 2660 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 07:09:46.832015 2120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:09:46.832131 2120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:09:46.832973 2120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5283f28bbbbf47789cdb632a6923f1ba: Sending DeleteTablet for 3 replicas of tablet a174dfc5f00e46f7b41c7327f5739916
I20260924 07:09:46.833189 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet a174dfc5f00e46f7b41c7327f5739916 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40000
I20260924 07:09:46.833201 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet a174dfc5f00e46f7b41c7327f5739916 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43970
I20260924 07:09:46.833204 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet a174dfc5f00e46f7b41c7327f5739916 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:46 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53366
I20260924 07:09:46.833431 2711 tablet_replica.cc:330] T a174dfc5f00e46f7b41c7327f5739916 P ae93f97d901b4fa0aba5531618b11c0c: stopping tablet replica
I20260924 07:09:46.833503 2711 raft_consensus.cc:2229] T a174dfc5f00e46f7b41c7327f5739916 P ae93f97d901b4fa0aba5531618b11c0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:09:46.833549 2711 raft_consensus.cc:2258] T a174dfc5f00e46f7b41c7327f5739916 P ae93f97d901b4fa0aba5531618b11c0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:09:46.833643 2712 tablet_replica.cc:330] T a174dfc5f00e46f7b41c7327f5739916 P a5cc149836284c468c0bff3680e18479: stopping tablet replica
I20260924 07:09:46.833704 2712 raft_consensus.cc:2229] T a174dfc5f00e46f7b41c7327f5739916 P a5cc149836284c468c0bff3680e18479 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:09:46.833767 2712 raft_consensus.cc:2258] T a174dfc5f00e46f7b41c7327f5739916 P a5cc149836284c468c0bff3680e18479 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:09:46.833865 2710 tablet_replica.cc:330] T a174dfc5f00e46f7b41c7327f5739916 P 9afadad6d73540b88766e90a1684b61f: stopping tablet replica
I20260924 07:09:46.833928 2710 raft_consensus.cc:2229] T a174dfc5f00e46f7b41c7327f5739916 P 9afadad6d73540b88766e90a1684b61f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:09:46.833971 2710 raft_consensus.cc:2258] T a174dfc5f00e46f7b41c7327f5739916 P 9afadad6d73540b88766e90a1684b61f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:09:46.834205 2712 ts_tablet_manager.cc:1905] T a174dfc5f00e46f7b41c7327f5739916 P a5cc149836284c468c0bff3680e18479: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:09:46.834240 2710 ts_tablet_manager.cc:1905] T a174dfc5f00e46f7b41c7327f5739916 P 9afadad6d73540b88766e90a1684b61f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:09:46.835407 2711 ts_tablet_manager.cc:1905] T a174dfc5f00e46f7b41c7327f5739916 P ae93f97d901b4fa0aba5531618b11c0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:09:46.836246 2710 ts_tablet_manager.cc:1918] T a174dfc5f00e46f7b41c7327f5739916 P 9afadad6d73540b88766e90a1684b61f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 07:09:46.836359 2710 log.cc:1196] T a174dfc5f00e46f7b41c7327f5739916 P 9afadad6d73540b88766e90a1684b61f: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-1/wal/wals/a174dfc5f00e46f7b41c7327f5739916
I20260924 07:09:46.836660 2710 ts_tablet_manager.cc:1939] T a174dfc5f00e46f7b41c7327f5739916 P 9afadad6d73540b88766e90a1684b61f: Deleting consensus metadata
I20260924 07:09:46.837009 2109 catalog_manager.cc:4841] TS 9afadad6d73540b88766e90a1684b61f (127.0.98.2:45009): tablet a174dfc5f00e46f7b41c7327f5739916 (table test-table [id=efd0fe2852184a88a20f8c71927dc961]) successfully deleted
I20260924 07:09:46.837332 2711 ts_tablet_manager.cc:1918] T a174dfc5f00e46f7b41c7327f5739916 P ae93f97d901b4fa0aba5531618b11c0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 07:09:46.837412 2711 log.cc:1196] T a174dfc5f00e46f7b41c7327f5739916 P ae93f97d901b4fa0aba5531618b11c0c: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-0/wal/wals/a174dfc5f00e46f7b41c7327f5739916
I20260924 07:09:46.837642 2711 ts_tablet_manager.cc:1939] T a174dfc5f00e46f7b41c7327f5739916 P ae93f97d901b4fa0aba5531618b11c0c: Deleting consensus metadata
I20260924 07:09:46.838042 2109 catalog_manager.cc:4841] TS ae93f97d901b4fa0aba5531618b11c0c (127.0.98.1:35995): tablet a174dfc5f00e46f7b41c7327f5739916 (table test-table [id=efd0fe2852184a88a20f8c71927dc961]) successfully deleted
I20260924 07:09:46.838522 2712 ts_tablet_manager.cc:1918] T a174dfc5f00e46f7b41c7327f5739916 P a5cc149836284c468c0bff3680e18479: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 07:09:46.838599 2712 log.cc:1196] T a174dfc5f00e46f7b41c7327f5739916 P a5cc149836284c468c0bff3680e18479: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-2/wal/wals/a174dfc5f00e46f7b41c7327f5739916
I20260924 07:09:46.838828 2712 ts_tablet_manager.cc:1939] T a174dfc5f00e46f7b41c7327f5739916 P a5cc149836284c468c0bff3680e18479: Deleting consensus metadata
I20260924 07:09:46.839269 2107 catalog_manager.cc:4841] TS a5cc149836284c468c0bff3680e18479 (127.0.98.3:37165): tablet a174dfc5f00e46f7b41c7327f5739916 (table test-table [id=efd0fe2852184a88a20f8c71927dc961]) successfully deleted
Sep 24 07:09:46 dist-test-slave-14ww krb5kdc[2077](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233786, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 07:09:46 dist-test-slave-14ww krb5kdc[2077](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233786, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:09:46.859197 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:46.849767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39744 (local address 127.0.98.62:37287)
0924 07:09:46.849889 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:46.849891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:46.849977 (+ 86us) server_negotiation.cc:407] Connection header received
0924 07:09:46.850019 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:46.850019 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:46.850032 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:46.850106 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:46.850633 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:46.851121 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:46.851857 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:46.852041 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:46.852091 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:46.854983 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:46.854985 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:46.854985 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:46.854992 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:46.858000 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:46.858479 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:46.858480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:46.858480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:46.858533 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:46.858702 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:46.858703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:46.858703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:46.858808 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:46.858884 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:46.859071 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 07:09:46.859132 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":80,"threads_started":1}
I20260924 07:09:46.862001 2120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:09:46.862149 2120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:09:46.863585 2259 tablet_service.cc:1467] Processing CreateTablet for tablet ea680419629f4bee9defec5a4d7ca768 (DEFAULT_TABLE table=test-table [id=0388a53fd4934eada90bae6542002a3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:46.863605 2399 tablet_service.cc:1467] Processing CreateTablet for tablet ea680419629f4bee9defec5a4d7ca768 (DEFAULT_TABLE table=test-table [id=0388a53fd4934eada90bae6542002a3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:46.863723 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea680419629f4bee9defec5a4d7ca768. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:46.863727 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea680419629f4bee9defec5a4d7ca768. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:46.864718 2721 tablet_bootstrap.cc:492] T ea680419629f4bee9defec5a4d7ca768 P ae93f97d901b4fa0aba5531618b11c0c: Bootstrap starting.
I20260924 07:09:46.864862 2539 tablet_service.cc:1467] Processing CreateTablet for tablet ea680419629f4bee9defec5a4d7ca768 (DEFAULT_TABLE table=test-table [id=0388a53fd4934eada90bae6542002a3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:46.864979 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea680419629f4bee9defec5a4d7ca768. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:46.865160 2722 tablet_bootstrap.cc:492] T ea680419629f4bee9defec5a4d7ca768 P 9afadad6d73540b88766e90a1684b61f: Bootstrap starting.
I20260924 07:09:46.865288 2721 tablet_bootstrap.cc:654] T ea680419629f4bee9defec5a4d7ca768 P ae93f97d901b4fa0aba5531618b11c0c: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:46.865541 2722 tablet_bootstrap.cc:654] T ea680419629f4bee9defec5a4d7ca768 P 9afadad6d73540b88766e90a1684b61f: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:46.865947 2721 tablet_bootstrap.cc:492] T ea680419629f4bee9defec5a4d7ca768 P ae93f97d901b4fa0aba5531618b11c0c: No bootstrap required, opened a new log
I20260924 07:09:46.866000 2721 ts_tablet_manager.cc:1397] T ea680419629f4bee9defec5a4d7ca768 P ae93f97d901b4fa0aba5531618b11c0c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 07:09:46.866134 2721 raft_consensus.cc:348] T ea680419629f4bee9defec5a4d7ca768 P ae93f97d901b4fa0aba5531618b11c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } }
I20260924 07:09:46.866216 2721 raft_consensus.cc:374] T ea680419629f4bee9defec5a4d7ca768 P ae93f97d901b4fa0aba5531618b11c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:46.866242 2721 raft_consensus.cc:729] T ea680419629f4bee9defec5a4d7ca768 P ae93f97d901b4fa0aba5531618b11c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae93f97d901b4fa0aba5531618b11c0c, State: Initialized, Role: FOLLOWER
I20260924 07:09:46.866317 2721 consensus_queue.cc:260] T ea680419629f4bee9defec5a4d7ca768 P ae93f97d901b4fa0aba5531618b11c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } }
I20260924 07:09:46.866441 2725 tablet_bootstrap.cc:492] T ea680419629f4bee9defec5a4d7ca768 P a5cc149836284c468c0bff3680e18479: Bootstrap starting.
I20260924 07:09:46.866509 2721 ts_tablet_manager.cc:1428] T ea680419629f4bee9defec5a4d7ca768 P ae93f97d901b4fa0aba5531618b11c0c: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 07:09:46.866878 2725 tablet_bootstrap.cc:654] T ea680419629f4bee9defec5a4d7ca768 P a5cc149836284c468c0bff3680e18479: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:46.866987 2722 tablet_bootstrap.cc:492] T ea680419629f4bee9defec5a4d7ca768 P 9afadad6d73540b88766e90a1684b61f: No bootstrap required, opened a new log
I20260924 07:09:46.867058 2722 ts_tablet_manager.cc:1397] T ea680419629f4bee9defec5a4d7ca768 P 9afadad6d73540b88766e90a1684b61f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:09:46.867229 2722 raft_consensus.cc:348] T ea680419629f4bee9defec5a4d7ca768 P 9afadad6d73540b88766e90a1684b61f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } }
I20260924 07:09:46.867318 2722 raft_consensus.cc:374] T ea680419629f4bee9defec5a4d7ca768 P 9afadad6d73540b88766e90a1684b61f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:46.867344 2722 raft_consensus.cc:729] T ea680419629f4bee9defec5a4d7ca768 P 9afadad6d73540b88766e90a1684b61f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9afadad6d73540b88766e90a1684b61f, State: Initialized, Role: FOLLOWER
I20260924 07:09:46.867412 2722 consensus_queue.cc:260] T ea680419629f4bee9defec5a4d7ca768 P 9afadad6d73540b88766e90a1684b61f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } }
I20260924 07:09:46.867558 2722 ts_tablet_manager.cc:1428] T ea680419629f4bee9defec5a4d7ca768 P 9afadad6d73540b88766e90a1684b61f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:46.867935 2725 tablet_bootstrap.cc:492] T ea680419629f4bee9defec5a4d7ca768 P a5cc149836284c468c0bff3680e18479: No bootstrap required, opened a new log
I20260924 07:09:46.868001 2725 ts_tablet_manager.cc:1397] T ea680419629f4bee9defec5a4d7ca768 P a5cc149836284c468c0bff3680e18479: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:09:46.868104 2725 raft_consensus.cc:348] T ea680419629f4bee9defec5a4d7ca768 P a5cc149836284c468c0bff3680e18479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } }
I20260924 07:09:46.868175 2725 raft_consensus.cc:374] T ea680419629f4bee9defec5a4d7ca768 P a5cc149836284c468c0bff3680e18479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:46.868196 2725 raft_consensus.cc:729] T ea680419629f4bee9defec5a4d7ca768 P a5cc149836284c468c0bff3680e18479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5cc149836284c468c0bff3680e18479, State: Initialized, Role: FOLLOWER
I20260924 07:09:46.868256 2725 consensus_queue.cc:260] T ea680419629f4bee9defec5a4d7ca768 P a5cc149836284c468c0bff3680e18479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } }
I20260924 07:09:46.868376 2725 ts_tablet_manager.cc:1428] T ea680419629f4bee9defec5a4d7ca768 P a5cc149836284c468c0bff3680e18479: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:47.021126 2726 raft_consensus.cc:482] T ea680419629f4bee9defec5a4d7ca768 P ae93f97d901b4fa0aba5531618b11c0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:09:47.021214 2726 raft_consensus.cc:504] T ea680419629f4bee9defec5a4d7ca768 P ae93f97d901b4fa0aba5531618b11c0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } }
I20260924 07:09:47.021598 2726 leader_election.cc:288] T ea680419629f4bee9defec5a4d7ca768 P ae93f97d901b4fa0aba5531618b11c0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9afadad6d73540b88766e90a1684b61f (127.0.98.2:45009), a5cc149836284c468c0bff3680e18479 (127.0.98.3:37165)
I20260924 07:09:47.024781 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:47.021927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43453 (local address 127.0.98.2:45009)
0924 07:09:47.022061 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:47.022062 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:47.022072 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:09:47.022143 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:47.022144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:47.022157 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:47.022222 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:47.022389 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:47.022914 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:47.023999 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:47.024377 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:47.024478 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:47.024490 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:47.024646 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 07:09:47.024685 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
I20260924 07:09:47.024771 2728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:47.021754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45009 (local address 127.0.98.1:43453)
0924 07:09:47.021993 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:47.022005 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:47.022066 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:47.022224 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:47.022224 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:47.022227 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:47.022303 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:47.022306 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:47.022998 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:47.022999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:47.023908 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:47.023911 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:47.024479 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:47.024480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:47.024554 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:47.024558 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:09:47.024620 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 07:09:47.024663 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":147,"threads_started":1}
I20260924 07:09:47.024873 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:47.021874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50003 (local address 127.0.98.3:37165)
0924 07:09:47.022005 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:47.022007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:47.022086 (+ 79us) server_negotiation.cc:407] Connection header received
0924 07:09:47.022159 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:47.022160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:47.022169 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:47.022264 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:47.022695 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:47.023203 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:47.024104 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:47.024476 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:47.024569 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:47.024579 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:47.024722 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 07:09:47.024788 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":77,"threads_started":1}
I20260924 07:09:47.024899 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:47.021823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37165 (local address 127.0.98.1:50003)
0924 07:09:47.022011 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:47.022021 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:47.022103 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:47.022280 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:47.022281 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:47.022283 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:47.022619 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:47.022621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:47.023302 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:47.023303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:47.024015 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:47.024018 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:47.024573 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:47.024573 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:47.024650 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:47.024653 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:09:47.024717 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 07:09:47.024782 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":99,"threads_started":1}
I20260924 07:09:47.025104 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea680419629f4bee9defec5a4d7ca768" candidate_uuid: "ae93f97d901b4fa0aba5531618b11c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5cc149836284c468c0bff3680e18479" is_pre_election: true
I20260924 07:09:47.025206 2559 raft_consensus.cc:2454] T ea680419629f4bee9defec5a4d7ca768 P a5cc149836284c468c0bff3680e18479 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae93f97d901b4fa0aba5531618b11c0c in term 0.
I20260924 07:09:47.025377 2211 leader_election.cc:302] T ea680419629f4bee9defec5a4d7ca768 P ae93f97d901b4fa0aba5531618b11c0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5cc149836284c468c0bff3680e18479, ae93f97d901b4fa0aba5531618b11c0c; no voters:
I20260924 07:09:47.025501 2726 raft_consensus.cc:2781] T ea680419629f4bee9defec5a4d7ca768 P ae93f97d901b4fa0aba5531618b11c0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:09:47.025540 2726 raft_consensus.cc:482] T ea680419629f4bee9defec5a4d7ca768 P ae93f97d901b4fa0aba5531618b11c0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:09:47.025563 2726 raft_consensus.cc:3037] T ea680419629f4bee9defec5a4d7ca768 P ae93f97d901b4fa0aba5531618b11c0c [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:47.025516 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea680419629f4bee9defec5a4d7ca768" candidate_uuid: "ae93f97d901b4fa0aba5531618b11c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9afadad6d73540b88766e90a1684b61f" is_pre_election: true
I20260924 07:09:47.025600 2419 raft_consensus.cc:2454] T ea680419629f4bee9defec5a4d7ca768 P 9afadad6d73540b88766e90a1684b61f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae93f97d901b4fa0aba5531618b11c0c in term 0.
I20260924 07:09:47.026244 2726 raft_consensus.cc:504] T ea680419629f4bee9defec5a4d7ca768 P ae93f97d901b4fa0aba5531618b11c0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } }
I20260924 07:09:47.026397 2726 leader_election.cc:288] T ea680419629f4bee9defec5a4d7ca768 P ae93f97d901b4fa0aba5531618b11c0c [CANDIDATE]: Term 1 election: Requested vote from peers 9afadad6d73540b88766e90a1684b61f (127.0.98.2:45009), a5cc149836284c468c0bff3680e18479 (127.0.98.3:37165)
I20260924 07:09:47.026494 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea680419629f4bee9defec5a4d7ca768" candidate_uuid: "ae93f97d901b4fa0aba5531618b11c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9afadad6d73540b88766e90a1684b61f"
I20260924 07:09:47.026574 2419 raft_consensus.cc:3037] T ea680419629f4bee9defec5a4d7ca768 P 9afadad6d73540b88766e90a1684b61f [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:47.026540 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea680419629f4bee9defec5a4d7ca768" candidate_uuid: "ae93f97d901b4fa0aba5531618b11c0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5cc149836284c468c0bff3680e18479"
I20260924 07:09:47.026605 2559 raft_consensus.cc:3037] T ea680419629f4bee9defec5a4d7ca768 P a5cc149836284c468c0bff3680e18479 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:47.027305 2559 raft_consensus.cc:2454] T ea680419629f4bee9defec5a4d7ca768 P a5cc149836284c468c0bff3680e18479 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae93f97d901b4fa0aba5531618b11c0c in term 1.
I20260924 07:09:47.027309 2419 raft_consensus.cc:2454] T ea680419629f4bee9defec5a4d7ca768 P 9afadad6d73540b88766e90a1684b61f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae93f97d901b4fa0aba5531618b11c0c in term 1.
I20260924 07:09:47.027458 2211 leader_election.cc:302] T ea680419629f4bee9defec5a4d7ca768 P ae93f97d901b4fa0aba5531618b11c0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5cc149836284c468c0bff3680e18479, ae93f97d901b4fa0aba5531618b11c0c; no voters:
I20260924 07:09:47.027567 2726 raft_consensus.cc:2781] T ea680419629f4bee9defec5a4d7ca768 P ae93f97d901b4fa0aba5531618b11c0c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:09:47.027696 2726 raft_consensus.cc:686] T ea680419629f4bee9defec5a4d7ca768 P ae93f97d901b4fa0aba5531618b11c0c [term 1 LEADER]: Becoming Leader. State: Replica: ae93f97d901b4fa0aba5531618b11c0c, State: Running, Role: LEADER
I20260924 07:09:47.027796 2726 consensus_queue.cc:237] T ea680419629f4bee9defec5a4d7ca768 P ae93f97d901b4fa0aba5531618b11c0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } }
I20260924 07:09:47.028501 2120 catalog_manager.cc:5495] T ea680419629f4bee9defec5a4d7ca768 P ae93f97d901b4fa0aba5531618b11c0c reported cstate change: term changed from 0 to 1, leader changed from <none> to ae93f97d901b4fa0aba5531618b11c0c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ae93f97d901b4fa0aba5531618b11c0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } health_report { overall_health: UNKNOWN } } }
I20260924 07:09:47.050573 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:47.048031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40002 (local address 127.0.98.1:35995)
0924 07:09:47.048179 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:47.048181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:47.048189 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:09:47.048230 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:47.048230 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:47.048247 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:47.048318 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:09:47.048454 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:47.049030 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:47.049899 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:47.050110 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:47.050162 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:47.050297 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:09:47.050392 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:09:47.050444 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:47.050484 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 07:09:47.050498 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":103,"threads_started":1}
I20260924 07:09:47.054970 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:47.052826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47909 (local address 127.0.98.2:45009)
0924 07:09:47.052882 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:47.052884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:47.053023 (+ 139us) server_negotiation.cc:407] Connection header received
0924 07:09:47.053060 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:47.053060 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:47.053070 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:47.053109 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:09:47.053219 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:47.053658 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:47.054343 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:47.054512 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:47.054570 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:47.054710 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:09:47.054734 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:09:47.054773 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:47.054894 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 07:09:47.054912 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 07:09:47.054996 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:47.052785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45009 (local address 127.0.98.1:47909)
0924 07:09:47.052956 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:47.052965 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:47.053015 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:47.053120 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:47.053121 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:47.053123 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:09:47.053167 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:47.053168 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:47.053719 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:47.053720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:47.054276 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:47.054278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:47.054583 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:47.054584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:47.054633 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:47.054644 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:09:47.054828 (+ 184us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:09:47.054831 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:09:47.054884 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 07:09:47.054921 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":99,"threads_started":1}
I20260924 07:09:47.056964 2559 raft_consensus.cc:1261] T ea680419629f4bee9defec5a4d7ca768 P a5cc149836284c468c0bff3680e18479 [term 1 FOLLOWER]: Refusing update from remote peer ae93f97d901b4fa0aba5531618b11c0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:09:47.057058 2419 raft_consensus.cc:1261] T ea680419629f4bee9defec5a4d7ca768 P 9afadad6d73540b88766e90a1684b61f [term 1 FOLLOWER]: Refusing update from remote peer ae93f97d901b4fa0aba5531618b11c0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:09:47.057243 2732 consensus_queue.cc:1035] T ea680419629f4bee9defec5a4d7ca768 P ae93f97d901b4fa0aba5531618b11c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:09:47.057366 2726 consensus_queue.cc:1035] T ea680419629f4bee9defec5a4d7ca768 P ae93f97d901b4fa0aba5531618b11c0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:09:47.064404 2735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:47.062111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45477 (local address 127.0.98.1:35995)
0924 07:09:47.062149 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:47.062151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:47.062257 (+ 106us) server_negotiation.cc:407] Connection header received
0924 07:09:47.062304 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:47.062304 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:47.062315 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:47.062374 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:09:47.062502 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:47.062999 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:47.063678 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:47.063903 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:47.063964 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:47.064186 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:09:47.064210 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:09:47.064249 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:47.064308 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 07:09:47.064328 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 07:09:47.064435 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:47.062058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35995 (local address 127.0.98.2:45477)
0924 07:09:47.062179 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:47.062190 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:47.062251 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:47.062379 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:47.062380 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:47.062386 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:09:47.062444 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:47.062446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:47.063071 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:47.063072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:47.063610 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:47.063611 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:47.063980 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:47.063981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:47.064046 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:47.064051 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:09:47.064259 (+ 208us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:09:47.064260 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 07:09:47.064305 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 07:09:47.064354 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":80,"threads_started":1}
I20260924 07:09:47.071182 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:09:47.071280 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:09:47.071908 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5283f28bbbbf47789cdb632a6923f1ba: Sending DeleteTablet for 3 replicas of tablet ea680419629f4bee9defec5a4d7ca768
I20260924 07:09:47.072149 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet ea680419629f4bee9defec5a4d7ca768 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53366
I20260924 07:09:47.072170 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet ea680419629f4bee9defec5a4d7ca768 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43970
I20260924 07:09:47.072242 2712 tablet_replica.cc:330] T ea680419629f4bee9defec5a4d7ca768 P a5cc149836284c468c0bff3680e18479: stopping tablet replica
I20260924 07:09:47.072292 2712 raft_consensus.cc:2229] T ea680419629f4bee9defec5a4d7ca768 P a5cc149836284c468c0bff3680e18479 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:09:47.072326 2712 raft_consensus.cc:2258] T ea680419629f4bee9defec5a4d7ca768 P a5cc149836284c468c0bff3680e18479 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:09:47.072276 2710 tablet_replica.cc:330] T ea680419629f4bee9defec5a4d7ca768 P 9afadad6d73540b88766e90a1684b61f: stopping tablet replica
I20260924 07:09:47.072376 2710 raft_consensus.cc:2229] T ea680419629f4bee9defec5a4d7ca768 P 9afadad6d73540b88766e90a1684b61f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:09:47.072412 2710 raft_consensus.cc:2258] T ea680419629f4bee9defec5a4d7ca768 P 9afadad6d73540b88766e90a1684b61f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:09:47.072363 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:09:47.072494 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:09:47.072626 2712 ts_tablet_manager.cc:1905] T ea680419629f4bee9defec5a4d7ca768 P a5cc149836284c468c0bff3680e18479: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:09:47.073036 2710 ts_tablet_manager.cc:1905] T ea680419629f4bee9defec5a4d7ca768 P 9afadad6d73540b88766e90a1684b61f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:09:47.074250 2712 ts_tablet_manager.cc:1918] T ea680419629f4bee9defec5a4d7ca768 P a5cc149836284c468c0bff3680e18479: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:09:47.074309 2712 log.cc:1196] T ea680419629f4bee9defec5a4d7ca768 P a5cc149836284c468c0bff3680e18479: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-2/wal/wals/ea680419629f4bee9defec5a4d7ca768
I20260924 07:09:47.074514 2712 ts_tablet_manager.cc:1939] T ea680419629f4bee9defec5a4d7ca768 P a5cc149836284c468c0bff3680e18479: Deleting consensus metadata
I20260924 07:09:47.074532 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet ea680419629f4bee9defec5a4d7ca768 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40000
I20260924 07:09:47.074615 2711 tablet_replica.cc:330] T ea680419629f4bee9defec5a4d7ca768 P ae93f97d901b4fa0aba5531618b11c0c: stopping tablet replica
I20260924 07:09:47.074661 2711 raft_consensus.cc:2229] T ea680419629f4bee9defec5a4d7ca768 P ae93f97d901b4fa0aba5531618b11c0c [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:09:47.074752 2399 tablet_service.cc:1467] Processing CreateTablet for tablet 58559ef7bf7743ab91818653f72a9410 (DEFAULT_TABLE table=test-table [id=1f66cf7c52e84cb8aa4871be141b669f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:47.074779 2711 raft_consensus.cc:2258] T ea680419629f4bee9defec5a4d7ca768 P ae93f97d901b4fa0aba5531618b11c0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:09:47.074771 2539 tablet_service.cc:1467] Processing CreateTablet for tablet 58559ef7bf7743ab91818653f72a9410 (DEFAULT_TABLE table=test-table [id=1f66cf7c52e84cb8aa4871be141b669f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:47.074862 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58559ef7bf7743ab91818653f72a9410. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:47.075007 2711 ts_tablet_manager.cc:1905] T ea680419629f4bee9defec5a4d7ca768 P ae93f97d901b4fa0aba5531618b11c0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:09:47.075186 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 58559ef7bf7743ab91818653f72a9410 (DEFAULT_TABLE table=test-table [id=1f66cf7c52e84cb8aa4871be141b669f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:47.075279 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58559ef7bf7743ab91818653f72a9410. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:47.076402 2711 ts_tablet_manager.cc:1918] T ea680419629f4bee9defec5a4d7ca768 P ae93f97d901b4fa0aba5531618b11c0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:09:47.076428 2725 tablet_bootstrap.cc:492] T 58559ef7bf7743ab91818653f72a9410 P a5cc149836284c468c0bff3680e18479: Bootstrap starting.
I20260924 07:09:47.076429 2721 tablet_bootstrap.cc:492] T 58559ef7bf7743ab91818653f72a9410 P ae93f97d901b4fa0aba5531618b11c0c: Bootstrap starting.
I20260924 07:09:47.076452 2711 log.cc:1196] T ea680419629f4bee9defec5a4d7ca768 P ae93f97d901b4fa0aba5531618b11c0c: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-0/wal/wals/ea680419629f4bee9defec5a4d7ca768
I20260924 07:09:47.076675 2711 ts_tablet_manager.cc:1939] T ea680419629f4bee9defec5a4d7ca768 P ae93f97d901b4fa0aba5531618b11c0c: Deleting consensus metadata
I20260924 07:09:47.076710 2725 tablet_bootstrap.cc:654] T 58559ef7bf7743ab91818653f72a9410 P a5cc149836284c468c0bff3680e18479: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:47.076853 2721 tablet_bootstrap.cc:654] T 58559ef7bf7743ab91818653f72a9410 P ae93f97d901b4fa0aba5531618b11c0c: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:47.077183 2109 catalog_manager.cc:4841] TS ae93f97d901b4fa0aba5531618b11c0c (127.0.98.1:35995): tablet ea680419629f4bee9defec5a4d7ca768 (table test-table [id=0388a53fd4934eada90bae6542002a3b]) successfully deleted
I20260924 07:09:47.077231 2710 ts_tablet_manager.cc:1918] T ea680419629f4bee9defec5a4d7ca768 P 9afadad6d73540b88766e90a1684b61f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:09:47.077278 2710 log.cc:1196] T ea680419629f4bee9defec5a4d7ca768 P 9afadad6d73540b88766e90a1684b61f: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-1/wal/wals/ea680419629f4bee9defec5a4d7ca768
I20260924 07:09:47.077286 2399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58559ef7bf7743ab91818653f72a9410. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:47.077414 2725 tablet_bootstrap.cc:492] T 58559ef7bf7743ab91818653f72a9410 P a5cc149836284c468c0bff3680e18479: No bootstrap required, opened a new log
I20260924 07:09:47.077456 2725 ts_tablet_manager.cc:1397] T 58559ef7bf7743ab91818653f72a9410 P a5cc149836284c468c0bff3680e18479: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:09:47.077512 2710 ts_tablet_manager.cc:1939] T ea680419629f4bee9defec5a4d7ca768 P 9afadad6d73540b88766e90a1684b61f: Deleting consensus metadata
I20260924 07:09:47.077557 2725 raft_consensus.cc:348] T 58559ef7bf7743ab91818653f72a9410 P a5cc149836284c468c0bff3680e18479 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } }
I20260924 07:09:47.077616 2725 raft_consensus.cc:374] T 58559ef7bf7743ab91818653f72a9410 P a5cc149836284c468c0bff3680e18479 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:47.077636 2725 raft_consensus.cc:729] T 58559ef7bf7743ab91818653f72a9410 P a5cc149836284c468c0bff3680e18479 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5cc149836284c468c0bff3680e18479, State: Initialized, Role: FOLLOWER
I20260924 07:09:47.077677 2725 consensus_queue.cc:260] T 58559ef7bf7743ab91818653f72a9410 P a5cc149836284c468c0bff3680e18479 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } }
I20260924 07:09:47.077759 2725 ts_tablet_manager.cc:1428] T 58559ef7bf7743ab91818653f72a9410 P a5cc149836284c468c0bff3680e18479: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:47.077941 2107 catalog_manager.cc:4841] TS a5cc149836284c468c0bff3680e18479 (127.0.98.3:37165): tablet ea680419629f4bee9defec5a4d7ca768 (table test-table [id=0388a53fd4934eada90bae6542002a3b]) successfully deleted
I20260924 07:09:47.078384 2721 tablet_bootstrap.cc:492] T 58559ef7bf7743ab91818653f72a9410 P ae93f97d901b4fa0aba5531618b11c0c: No bootstrap required, opened a new log
I20260924 07:09:47.078439 2721 ts_tablet_manager.cc:1397] T 58559ef7bf7743ab91818653f72a9410 P ae93f97d901b4fa0aba5531618b11c0c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 07:09:47.078567 2721 raft_consensus.cc:348] T 58559ef7bf7743ab91818653f72a9410 P ae93f97d901b4fa0aba5531618b11c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } }
I20260924 07:09:47.078653 2721 raft_consensus.cc:374] T 58559ef7bf7743ab91818653f72a9410 P ae93f97d901b4fa0aba5531618b11c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:47.078680 2721 raft_consensus.cc:729] T 58559ef7bf7743ab91818653f72a9410 P ae93f97d901b4fa0aba5531618b11c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae93f97d901b4fa0aba5531618b11c0c, State: Initialized, Role: FOLLOWER
I20260924 07:09:47.078694 2109 catalog_manager.cc:4841] TS 9afadad6d73540b88766e90a1684b61f (127.0.98.2:45009): tablet ea680419629f4bee9defec5a4d7ca768 (table test-table [id=0388a53fd4934eada90bae6542002a3b]) successfully deleted
I20260924 07:09:47.078716 2721 consensus_queue.cc:260] T 58559ef7bf7743ab91818653f72a9410 P ae93f97d901b4fa0aba5531618b11c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } }
I20260924 07:09:47.078830 2721 ts_tablet_manager.cc:1428] T 58559ef7bf7743ab91818653f72a9410 P ae93f97d901b4fa0aba5531618b11c0c: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 07:09:47.078929 2722 tablet_bootstrap.cc:492] T 58559ef7bf7743ab91818653f72a9410 P 9afadad6d73540b88766e90a1684b61f: Bootstrap starting.
I20260924 07:09:47.079253 2722 tablet_bootstrap.cc:654] T 58559ef7bf7743ab91818653f72a9410 P 9afadad6d73540b88766e90a1684b61f: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:47.079887 2722 tablet_bootstrap.cc:492] T 58559ef7bf7743ab91818653f72a9410 P 9afadad6d73540b88766e90a1684b61f: No bootstrap required, opened a new log
I20260924 07:09:47.079939 2722 ts_tablet_manager.cc:1397] T 58559ef7bf7743ab91818653f72a9410 P 9afadad6d73540b88766e90a1684b61f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:09:47.080040 2722 raft_consensus.cc:348] T 58559ef7bf7743ab91818653f72a9410 P 9afadad6d73540b88766e90a1684b61f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } }
I20260924 07:09:47.080119 2722 raft_consensus.cc:374] T 58559ef7bf7743ab91818653f72a9410 P 9afadad6d73540b88766e90a1684b61f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:47.080140 2722 raft_consensus.cc:729] T 58559ef7bf7743ab91818653f72a9410 P 9afadad6d73540b88766e90a1684b61f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9afadad6d73540b88766e90a1684b61f, State: Initialized, Role: FOLLOWER
I20260924 07:09:47.080181 2722 consensus_queue.cc:260] T 58559ef7bf7743ab91818653f72a9410 P 9afadad6d73540b88766e90a1684b61f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } }
I20260924 07:09:47.080268 2722 ts_tablet_manager.cc:1428] T 58559ef7bf7743ab91818653f72a9410 P 9afadad6d73540b88766e90a1684b61f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:47.095460 2688 raft_consensus.cc:482] T 58559ef7bf7743ab91818653f72a9410 P 9afadad6d73540b88766e90a1684b61f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:09:47.095542 2688 raft_consensus.cc:504] T 58559ef7bf7743ab91818653f72a9410 P 9afadad6d73540b88766e90a1684b61f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } }
I20260924 07:09:47.095693 2688 leader_election.cc:288] T 58559ef7bf7743ab91818653f72a9410 P 9afadad6d73540b88766e90a1684b61f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae93f97d901b4fa0aba5531618b11c0c (127.0.98.1:35995), a5cc149836284c468c0bff3680e18479 (127.0.98.3:37165)
I20260924 07:09:47.095808 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58559ef7bf7743ab91818653f72a9410" candidate_uuid: "9afadad6d73540b88766e90a1684b61f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae93f97d901b4fa0aba5531618b11c0c" is_pre_election: true
I20260924 07:09:47.095920 2279 raft_consensus.cc:2454] T 58559ef7bf7743ab91818653f72a9410 P ae93f97d901b4fa0aba5531618b11c0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9afadad6d73540b88766e90a1684b61f in term 0.
I20260924 07:09:47.095901 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58559ef7bf7743ab91818653f72a9410" candidate_uuid: "9afadad6d73540b88766e90a1684b61f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5cc149836284c468c0bff3680e18479" is_pre_election: true
I20260924 07:09:47.095974 2559 raft_consensus.cc:2454] T 58559ef7bf7743ab91818653f72a9410 P a5cc149836284c468c0bff3680e18479 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9afadad6d73540b88766e90a1684b61f in term 0.
I20260924 07:09:47.096067 2353 leader_election.cc:302] T 58559ef7bf7743ab91818653f72a9410 P 9afadad6d73540b88766e90a1684b61f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9afadad6d73540b88766e90a1684b61f, ae93f97d901b4fa0aba5531618b11c0c; no voters:
I20260924 07:09:47.096170 2688 raft_consensus.cc:2781] T 58559ef7bf7743ab91818653f72a9410 P 9afadad6d73540b88766e90a1684b61f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:09:47.096210 2688 raft_consensus.cc:482] T 58559ef7bf7743ab91818653f72a9410 P 9afadad6d73540b88766e90a1684b61f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:09:47.096225 2688 raft_consensus.cc:3037] T 58559ef7bf7743ab91818653f72a9410 P 9afadad6d73540b88766e90a1684b61f [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:47.096977 2688 raft_consensus.cc:504] T 58559ef7bf7743ab91818653f72a9410 P 9afadad6d73540b88766e90a1684b61f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } }
I20260924 07:09:47.097110 2688 leader_election.cc:288] T 58559ef7bf7743ab91818653f72a9410 P 9afadad6d73540b88766e90a1684b61f [CANDIDATE]: Term 1 election: Requested vote from peers ae93f97d901b4fa0aba5531618b11c0c (127.0.98.1:35995), a5cc149836284c468c0bff3680e18479 (127.0.98.3:37165)
I20260924 07:09:47.097245 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58559ef7bf7743ab91818653f72a9410" candidate_uuid: "9afadad6d73540b88766e90a1684b61f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5cc149836284c468c0bff3680e18479"
I20260924 07:09:47.097272 2279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58559ef7bf7743ab91818653f72a9410" candidate_uuid: "9afadad6d73540b88766e90a1684b61f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae93f97d901b4fa0aba5531618b11c0c"
I20260924 07:09:47.097321 2559 raft_consensus.cc:3037] T 58559ef7bf7743ab91818653f72a9410 P a5cc149836284c468c0bff3680e18479 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:47.097339 2279 raft_consensus.cc:3037] T 58559ef7bf7743ab91818653f72a9410 P ae93f97d901b4fa0aba5531618b11c0c [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:47.098029 2279 raft_consensus.cc:2454] T 58559ef7bf7743ab91818653f72a9410 P ae93f97d901b4fa0aba5531618b11c0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9afadad6d73540b88766e90a1684b61f in term 1.
I20260924 07:09:47.098029 2559 raft_consensus.cc:2454] T 58559ef7bf7743ab91818653f72a9410 P a5cc149836284c468c0bff3680e18479 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9afadad6d73540b88766e90a1684b61f in term 1.
I20260924 07:09:47.098192 2351 leader_election.cc:302] T 58559ef7bf7743ab91818653f72a9410 P 9afadad6d73540b88766e90a1684b61f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9afadad6d73540b88766e90a1684b61f, a5cc149836284c468c0bff3680e18479; no voters:
I20260924 07:09:47.098304 2688 raft_consensus.cc:2781] T 58559ef7bf7743ab91818653f72a9410 P 9afadad6d73540b88766e90a1684b61f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:09:47.098371 2688 raft_consensus.cc:686] T 58559ef7bf7743ab91818653f72a9410 P 9afadad6d73540b88766e90a1684b61f [term 1 LEADER]: Becoming Leader. State: Replica: 9afadad6d73540b88766e90a1684b61f, State: Running, Role: LEADER
I20260924 07:09:47.098423 2688 consensus_queue.cc:237] T 58559ef7bf7743ab91818653f72a9410 P 9afadad6d73540b88766e90a1684b61f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } }
I20260924 07:09:47.098914 2121 catalog_manager.cc:5495] T 58559ef7bf7743ab91818653f72a9410 P 9afadad6d73540b88766e90a1684b61f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9afadad6d73540b88766e90a1684b61f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "9afadad6d73540b88766e90a1684b61f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9afadad6d73540b88766e90a1684b61f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 } health_report { overall_health: UNKNOWN } } }
I20260924 07:09:47.106076 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:47.103563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43980 (local address 127.0.98.2:45009)
0924 07:09:47.103607 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:47.103609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:47.103643 (+ 34us) server_negotiation.cc:407] Connection header received
0924 07:09:47.103668 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:47.103669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:47.103679 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:47.103735 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:09:47.103819 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:47.104587 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:47.105344 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:47.105659 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:47.105732 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:47.105846 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:09:47.105880 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:09:47.105930 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:47.105975 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 07:09:47.105995 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 07:09:47.110606 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:47.108119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45009 (local address 127.0.98.2:41625)
0924 07:09:47.108172 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:47.108183 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:47.108218 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:47.108395 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:47.108395 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:47.108396 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:09:47.108425 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:47.108426 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:47.109268 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:47.109268 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:47.109782 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:47.109783 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:47.110239 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:47.110240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:47.110275 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:47.110279 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:09:47.110439 (+ 160us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:09:47.110441 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 07:09:47.110496 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 07:09:47.110530 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 07:09:47.110651 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:47.108267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41625 (local address 127.0.98.2:45009)
0924 07:09:47.108299 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:47.108300 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:47.108307 (+ 7us) server_negotiation.cc:407] Connection header received
0924 07:09:47.108327 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:47.108328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:47.108336 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:47.108393 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:09:47.108480 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:47.109198 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:47.109862 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:47.110167 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:47.110236 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:47.110329 (+ 93us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:09:47.110357 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:09:47.110402 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:47.110512 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 07:09:47.110527 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 07:09:47.112514 2559 raft_consensus.cc:1261] T 58559ef7bf7743ab91818653f72a9410 P a5cc149836284c468c0bff3680e18479 [term 1 FOLLOWER]: Refusing update from remote peer 9afadad6d73540b88766e90a1684b61f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:09:47.112504 2279 raft_consensus.cc:1261] T 58559ef7bf7743ab91818653f72a9410 P ae93f97d901b4fa0aba5531618b11c0c [term 1 FOLLOWER]: Refusing update from remote peer 9afadad6d73540b88766e90a1684b61f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:09:47.112709 2696 consensus_queue.cc:1035] T 58559ef7bf7743ab91818653f72a9410 P 9afadad6d73540b88766e90a1684b61f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae93f97d901b4fa0aba5531618b11c0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:09:47.112835 2691 consensus_queue.cc:1035] T 58559ef7bf7743ab91818653f72a9410 P 9afadad6d73540b88766e90a1684b61f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5cc149836284c468c0bff3680e18479" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 07:09:47.117446 2689 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 07:09:48.635975 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:09:48.636101 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:09:48.636888 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5283f28bbbbf47789cdb632a6923f1ba: Sending DeleteTablet for 3 replicas of tablet 58559ef7bf7743ab91818653f72a9410
I20260924 07:09:48.637143 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet 58559ef7bf7743ab91818653f72a9410 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53366
I20260924 07:09:48.637171 2399 tablet_service.cc:1514] Processing DeleteTablet for tablet 58559ef7bf7743ab91818653f72a9410 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43970
I20260924 07:09:48.637262 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 2194
I20260924 07:09:48.637312 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet 58559ef7bf7743ab91818653f72a9410 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40000
I20260924 07:09:48.639652 2773 tablet_replica.cc:330] T 58559ef7bf7743ab91818653f72a9410 P a5cc149836284c468c0bff3680e18479: stopping tablet replica
I20260924 07:09:48.639731 2773 raft_consensus.cc:2229] T 58559ef7bf7743ab91818653f72a9410 P a5cc149836284c468c0bff3680e18479 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:09:48.639770 2773 raft_consensus.cc:2258] T 58559ef7bf7743ab91818653f72a9410 P a5cc149836284c468c0bff3680e18479 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:09:48.639885 2774 tablet_replica.cc:330] T 58559ef7bf7743ab91818653f72a9410 P 9afadad6d73540b88766e90a1684b61f: stopping tablet replica
I20260924 07:09:48.639943 2774 raft_consensus.cc:2229] T 58559ef7bf7743ab91818653f72a9410 P 9afadad6d73540b88766e90a1684b61f [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:09:48.640025 2774 raft_consensus.cc:2258] T 58559ef7bf7743ab91818653f72a9410 P 9afadad6d73540b88766e90a1684b61f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:09:48.640033 2773 ts_tablet_manager.cc:1905] T 58559ef7bf7743ab91818653f72a9410 P a5cc149836284c468c0bff3680e18479: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:09:48.640290 2774 ts_tablet_manager.cc:1905] T 58559ef7bf7743ab91818653f72a9410 P 9afadad6d73540b88766e90a1684b61f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:09:48.641685 2773 ts_tablet_manager.cc:1918] T 58559ef7bf7743ab91818653f72a9410 P a5cc149836284c468c0bff3680e18479: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 07:09:48.641762 2773 log.cc:1196] T 58559ef7bf7743ab91818653f72a9410 P a5cc149836284c468c0bff3680e18479: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-2/wal/wals/58559ef7bf7743ab91818653f72a9410
I20260924 07:09:48.641966 2773 ts_tablet_manager.cc:1939] T 58559ef7bf7743ab91818653f72a9410 P a5cc149836284c468c0bff3680e18479: Deleting consensus metadata
I20260924 07:09:48.642273 2774 ts_tablet_manager.cc:1918] T 58559ef7bf7743ab91818653f72a9410 P 9afadad6d73540b88766e90a1684b61f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 07:09:48.642295 2107 catalog_manager.cc:4841] TS a5cc149836284c468c0bff3680e18479 (127.0.98.3:37165): tablet 58559ef7bf7743ab91818653f72a9410 (table test-table [id=1f66cf7c52e84cb8aa4871be141b669f]) successfully deleted
I20260924 07:09:48.642334 2774 log.cc:1196] T 58559ef7bf7743ab91818653f72a9410 P 9afadad6d73540b88766e90a1684b61f: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790233772407728-392-0/minicluster-data/ts-1/wal/wals/58559ef7bf7743ab91818653f72a9410
I20260924 07:09:48.642570 2774 ts_tablet_manager.cc:1939] T 58559ef7bf7743ab91818653f72a9410 P 9afadad6d73540b88766e90a1684b61f: Deleting consensus metadata
I20260924 07:09:48.642853 2109 catalog_manager.cc:4841] TS 9afadad6d73540b88766e90a1684b61f (127.0.98.2:45009): tablet 58559ef7bf7743ab91818653f72a9410 (table test-table [id=1f66cf7c52e84cb8aa4871be141b669f]) successfully deleted
W20260924 07:09:48.644454 2109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35995 (error 108)
I20260924 07:09:48.644686 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 2334
I20260924 07:09:48.647112 2776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:48.644884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35995 (local address 127.0.0.1:40016)
0924 07:09:48.647017 (+ 2133us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:48.647066 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2107,"thread_start_us":66,"threads_started":1}
W20260924 07:09:48.647246 2109 catalog_manager.cc:4584] TS ae93f97d901b4fa0aba5531618b11c0c (127.0.98.1:35995): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 58559ef7bf7743ab91818653f72a9410: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35995: connect: Connection refused (error 111)
I20260924 07:09:48.650687 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 2474
I20260924 07:09:48.655761 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 2090
2026-09-24T07:09:48Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5895 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:09:48 dist-test-slave-14ww krb5kdc[2779](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:09:48 dist-test-slave-14ww krb5kdc[2779](info): set up 2 sockets
Sep 24 07:09:48 dist-test-slave-14ww krb5kdc[2779](info): commencing operation
krb5kdc: starting...
W20260924 07:09:49.700433 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:09:49 dist-test-slave-14ww krb5kdc[2779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:09:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:09:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:09:49.820680 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45911
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45911
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:49.900689 2792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:49.900980 2792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:49.901015 2792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:49.902191 2792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:09:49.902256 2792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:49.902281 2792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:49.902308 2792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:09:49.902340 2792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:09:49.903831 2792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:49.904110 2792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:49.904333 2792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:49.909243 2800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:49.909339 2797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:49.909299 2798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:49.909533 2792 server_base.cc:1034] running on GCE node
I20260924 07:09:49.909778 2792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:49.910054 2792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:49.911211 2792 hybrid_clock.cc:648] HybridClock initialized: now 1790233789911198 us; error 46 us; skew 500 ppm
Sep 24 07:09:49 dist-test-slave-14ww krb5kdc[2779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233789, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:49.913722 2792 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:09:49.914142 2792 webserver.cc:460] Webserver started at http://127.0.98.62:37079/ using document root <none> and password file <none>
I20260924 07:09:49.914357 2792 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:49.914423 2792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:49.914583 2792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:49.915604 2792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "8808944aef50417abb80e36146ce45bb"
format_stamp: "Formatted at 2026-09-24 07:09:49 on dist-test-slave-14ww"
server_key: "075ca1e0b39d32f608a6a1eeee6a8ee0"
server_key_iv: "4e3ed9b9d817e2c8210e8d23e3fc7da4"
server_key_version: "encryptionkey@0"
I20260924 07:09:49.915957 2792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "8808944aef50417abb80e36146ce45bb"
format_stamp: "Formatted at 2026-09-24 07:09:49 on dist-test-slave-14ww"
server_key: "075ca1e0b39d32f608a6a1eeee6a8ee0"
server_key_iv: "4e3ed9b9d817e2c8210e8d23e3fc7da4"
server_key_version: "encryptionkey@0"
I20260924 07:09:49.917326 2792 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:09:49.918121 2807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:49.918304 2792 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:09:49.918432 2792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "8808944aef50417abb80e36146ce45bb"
format_stamp: "Formatted at 2026-09-24 07:09:49 on dist-test-slave-14ww"
server_key: "075ca1e0b39d32f608a6a1eeee6a8ee0"
server_key_iv: "4e3ed9b9d817e2c8210e8d23e3fc7da4"
server_key_version: "encryptionkey@0"
I20260924 07:09:49.918509 2792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:49.936939 2792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:49.939329 2792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:49.939496 2792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:49.943351 2792 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45911
I20260924 07:09:49.943405 2859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45911 every 8 connection(s)
I20260924 07:09:49.943907 2792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:09:49.944542 2860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:49.945147 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 2792
I20260924 07:09:49.945250 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:09:49.945439 392 external_mini_cluster.cc:1428] Setting key 2d768bca99b718dc228c8bc4c440a4ca
I20260924 07:09:49.947896 2860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8808944aef50417abb80e36146ce45bb: Bootstrap starting.
I20260924 07:09:49.948611 2860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8808944aef50417abb80e36146ce45bb: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:49.948992 2860 log.cc:824] T 00000000000000000000000000000000 P 8808944aef50417abb80e36146ce45bb: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:49.949753 2860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8808944aef50417abb80e36146ce45bb: No bootstrap required, opened a new log
I20260924 07:09:49.952361 2860 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8808944aef50417abb80e36146ce45bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8808944aef50417abb80e36146ce45bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45911 } }
I20260924 07:09:49.952515 2860 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8808944aef50417abb80e36146ce45bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:49.952545 2860 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8808944aef50417abb80e36146ce45bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8808944aef50417abb80e36146ce45bb, State: Initialized, Role: FOLLOWER
I20260924 07:09:49.952656 2860 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8808944aef50417abb80e36146ce45bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8808944aef50417abb80e36146ce45bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45911 } }
I20260924 07:09:49.952793 2860 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8808944aef50417abb80e36146ce45bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:09:49.952831 2860 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8808944aef50417abb80e36146ce45bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:09:49.952876 2860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8808944aef50417abb80e36146ce45bb [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:49.953531 2860 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8808944aef50417abb80e36146ce45bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8808944aef50417abb80e36146ce45bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45911 } }
I20260924 07:09:49.953614 2860 leader_election.cc:302] T 00000000000000000000000000000000 P 8808944aef50417abb80e36146ce45bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8808944aef50417abb80e36146ce45bb; no voters:
I20260924 07:09:49.953728 2860 leader_election.cc:288] T 00000000000000000000000000000000 P 8808944aef50417abb80e36146ce45bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:09:49.953814 2865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8808944aef50417abb80e36146ce45bb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:09:49.953948 2860 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8808944aef50417abb80e36146ce45bb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:09:49.953984 2865 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8808944aef50417abb80e36146ce45bb [term 1 LEADER]: Becoming Leader. State: Replica: 8808944aef50417abb80e36146ce45bb, State: Running, Role: LEADER
I20260924 07:09:49.954140 2865 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8808944aef50417abb80e36146ce45bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8808944aef50417abb80e36146ce45bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45911 } }
I20260924 07:09:49.955602 2866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8808944aef50417abb80e36146ce45bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8808944aef50417abb80e36146ce45bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8808944aef50417abb80e36146ce45bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45911 } } }
I20260924 07:09:49.955727 2866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8808944aef50417abb80e36146ce45bb [sys.catalog]: This master's current role is: LEADER
I20260924 07:09:49.955750 2867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8808944aef50417abb80e36146ce45bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8808944aef50417abb80e36146ce45bb. Latest consensus state: current_term: 1 leader_uuid: "8808944aef50417abb80e36146ce45bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8808944aef50417abb80e36146ce45bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45911 } } }
I20260924 07:09:49.955798 2867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8808944aef50417abb80e36146ce45bb [sys.catalog]: This master's current role is: LEADER
W20260924 07:09:49.956192 2880 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8808944aef50417abb80e36146ce45bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:09:49.956249 2880 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:09:49.956303 2881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:09:49.956463 2881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 07:09:49 dist-test-slave-14ww krb5kdc[2779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233789, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:09:49.957958 2881 catalog_manager.cc:1347] Generated new cluster ID: 2df3c9f3b69a4fba8eddc3c4a34431ac
I20260924 07:09:49.958014 2881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:09:49.961272 2863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:49.945992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45274 (local address 127.0.98.62:45911)
0924 07:09:49.946184 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:49.946193 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:49.949433 (+ 3240us) server_negotiation.cc:407] Connection header received
0924 07:09:49.950722 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:49.950731 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:49.950806 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:49.950970 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:49.951778 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:49.953254 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:49.954307 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:49.955108 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:49.955221 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:49.958229 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:49.958232 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:49.958232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:49.958241 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:49.959888 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:49.960242 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:49.960244 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:49.960245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:49.960322 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:49.960500 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:49.960501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:49.960501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:49.960696 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:49.960881 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:49.960953 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 07:09:49.961086 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":87,"threads_started":1}
I20260924 07:09:49.976936 2881 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:09:49.977448 2881 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:09:49.980496 2881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8808944aef50417abb80e36146ce45bb: Generated new TSK 0
I20260924 07:09:49.980655 2881 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 07:09:50.012027 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45911
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:50.091228 2886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:50.091411 2886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:50.091434 2886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:50.092568 2886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:50.092609 2886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:50.092660 2886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:09:50.094126 2886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:50.094341 2886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:50.094579 2886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:50.099704 2891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:50.099725 2892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:50.099828 2886 server_base.cc:1034] running on GCE node
W20260924 07:09:50.099737 2894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:50.100175 2886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:50.100363 2886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:50.101522 2886 hybrid_clock.cc:648] HybridClock initialized: now 1790233790101503 us; error 32 us; skew 500 ppm
Sep 24 07:09:50 dist-test-slave-14ww krb5kdc[2779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:50.103943 2886 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:09:50.104334 2886 webserver.cc:460] Webserver started at http://127.0.98.1:36645/ using document root <none> and password file <none>
I20260924 07:09:50.104583 2886 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:50.104650 2886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:50.104813 2886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:50.105875 2886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-0/data/instance:
uuid: "0c94babda8c34a9193a38c1eb01109ee"
format_stamp: "Formatted at 2026-09-24 07:09:50 on dist-test-slave-14ww"
server_key: "7fa0235a32fc1eade76f23cebcbf13f2"
server_key_iv: "2f5f9ad7eff98161fd02e45964dc2c92"
server_key_version: "encryptionkey@0"
I20260924 07:09:50.106236 2886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0c94babda8c34a9193a38c1eb01109ee"
format_stamp: "Formatted at 2026-09-24 07:09:50 on dist-test-slave-14ww"
server_key: "7fa0235a32fc1eade76f23cebcbf13f2"
server_key_iv: "2f5f9ad7eff98161fd02e45964dc2c92"
server_key_version: "encryptionkey@0"
I20260924 07:09:50.107474 2886 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 07:09:50.108124 2901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:50.108270 2886 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:09:50.108358 2886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-0/wal
uuid: "0c94babda8c34a9193a38c1eb01109ee"
format_stamp: "Formatted at 2026-09-24 07:09:50 on dist-test-slave-14ww"
server_key: "7fa0235a32fc1eade76f23cebcbf13f2"
server_key_iv: "2f5f9ad7eff98161fd02e45964dc2c92"
server_key_version: "encryptionkey@0"
I20260924 07:09:50.108435 2886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:50.115409 2886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:50.117628 2886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:50.117779 2886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:50.117961 2886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:50.118176 2886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:50.118212 2886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:50.118232 2886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:50.118244 2886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:50.124002 2886 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45745
I20260924 07:09:50.124045 3014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45745 every 8 connection(s)
I20260924 07:09:50.124312 2886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:09:50.126492 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 2886
I20260924 07:09:50.126591 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:09:50.126755 392 external_mini_cluster.cc:1428] Setting key 558a097018d63487cd4509e4969539d8
Sep 24 07:09:50 dist-test-slave-14ww krb5kdc[2779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:09:50.134716 2863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:50.124979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37879 (local address 127.0.98.62:45911)
0924 07:09:50.125020 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:50.125022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:50.125244 (+ 222us) server_negotiation.cc:407] Connection header received
0924 07:09:50.126189 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:50.126190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:50.126202 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:50.126256 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:50.127036 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.127605 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.129087 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.129299 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.129380 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:50.131847 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:50.131848 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:50.131849 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:50.131856 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:50.133289 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:50.133789 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:50.133790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:50.133790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:50.133841 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:50.134106 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:50.134107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:50.134107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:50.134200 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:50.134248 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:50.134581 (+ 333us) server_negotiation.cc:299] Negotiation successful
0924 07:09:50.134651 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 07:09:50.134984 3017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:50.124954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45911 (local address 127.0.98.1:37879)
0924 07:09:50.125120 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:50.125152 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:50.126075 (+ 923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:50.126321 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:50.126325 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:50.126686 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:50.126949 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:50.126954 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.127694 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.127695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:50.128995 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:50.128998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.129385 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.129387 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:50.129515 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:50.130072 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:50.130080 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:50.131738 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:50.133374 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:50.133381 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:50.133383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:50.133700 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:50.133911 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:50.133912 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:50.133913 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:50.134038 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:50.134267 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:50.134274 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:50.134438 (+ 164us) client_negotiation.cc:769] Sending connection context
0924 07:09:50.134563 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 07:09:50.134735 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":76,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 07:09:50.135556 3015 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45911
I20260924 07:09:50.135658 3015 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:50.135838 3015 heartbeater.cc:511] Master 127.0.98.62:45911 requested a full tablet report, sending...
I20260924 07:09:50.136346 2824 ts_manager.cc:194] Registered new tserver with Master: 0c94babda8c34a9193a38c1eb01109ee (127.0.98.1:45745)
I20260924 07:09:50.137055 2824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37879
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 07:09:50.152381 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45911
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:50.231850 3020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:50.232081 3020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:50.232110 3020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:50.233227 3020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:50.233264 3020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:50.233289 3020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:09:50.234894 3020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:50.235159 3020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:50.235405 3020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:50.241146 3025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:50.241155 3028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:50.241146 3026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:50.241232 3020 server_base.cc:1034] running on GCE node
I20260924 07:09:50.241557 3020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:50.241747 3020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:50.242906 3020 hybrid_clock.cc:648] HybridClock initialized: now 1790233790242885 us; error 33 us; skew 500 ppm
Sep 24 07:09:50 dist-test-slave-14ww krb5kdc[2779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:50.245517 3020 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:09:50.245946 3020 webserver.cc:460] Webserver started at http://127.0.98.2:39283/ using document root <none> and password file <none>
I20260924 07:09:50.246121 3020 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:50.246160 3020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:50.246243 3020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:50.247301 3020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-1/data/instance:
uuid: "ca3c698d8d4743ed95801d9714a751d7"
format_stamp: "Formatted at 2026-09-24 07:09:50 on dist-test-slave-14ww"
server_key: "91fc348c28173313f53e483fbf684c1c"
server_key_iv: "3adbcb487465a2b112fc49871bb97599"
server_key_version: "encryptionkey@0"
I20260924 07:09:50.247611 3020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ca3c698d8d4743ed95801d9714a751d7"
format_stamp: "Formatted at 2026-09-24 07:09:50 on dist-test-slave-14ww"
server_key: "91fc348c28173313f53e483fbf684c1c"
server_key_iv: "3adbcb487465a2b112fc49871bb97599"
server_key_version: "encryptionkey@0"
I20260924 07:09:50.248852 3020 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:09:50.249507 3035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:50.249689 3020 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:09:50.249778 3020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-1/wal
uuid: "ca3c698d8d4743ed95801d9714a751d7"
format_stamp: "Formatted at 2026-09-24 07:09:50 on dist-test-slave-14ww"
server_key: "91fc348c28173313f53e483fbf684c1c"
server_key_iv: "3adbcb487465a2b112fc49871bb97599"
server_key_version: "encryptionkey@0"
I20260924 07:09:50.249831 3020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:50.269836 3020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:50.271610 3020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:50.271716 3020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:50.271888 3020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:50.272179 3020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:50.272212 3020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:50.272233 3020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:50.272281 3020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:50.277719 3020 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39777
I20260924 07:09:50.277797 3148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39777 every 8 connection(s)
I20260924 07:09:50.278103 3020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 07:09:50 dist-test-slave-14ww krb5kdc[2779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:09:50.286989 2863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:50.278900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55179 (local address 127.0.98.62:45911)
0924 07:09:50.278960 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:50.278962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:50.279189 (+ 227us) server_negotiation.cc:407] Connection header received
0924 07:09:50.280078 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:50.280079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:50.280088 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:50.280145 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:50.280704 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.281253 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.282265 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.282484 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.282553 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:50.284552 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:50.284554 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:50.284554 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:50.284560 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:50.285840 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:50.286218 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:50.286219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:50.286220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:50.286276 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:50.286487 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:50.286487 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:50.286488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:50.286587 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:50.286640 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:50.286848 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 07:09:50.286925 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 07:09:50.287075 3151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:50.278857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45911 (local address 127.0.98.2:55179)
0924 07:09:50.279058 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:50.279098 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:50.279991 (+ 893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:50.280196 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:50.280199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:50.280457 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:50.280632 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:50.280636 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.281359 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.281360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:50.282167 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:50.282169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.282581 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.282582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:50.282684 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:50.283182 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:50.283190 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:50.284468 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:50.285928 (+ 1460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:50.285933 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:50.285934 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:50.286137 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:50.286343 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:50.286343 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:50.286343 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:50.286431 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:50.286640 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:50.286643 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:50.286744 (+ 101us) client_negotiation.cc:769] Sending connection context
0924 07:09:50.286820 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 07:09:50.286928 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":128,"threads_started":1}
I20260924 07:09:50.287308 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 3020
I20260924 07:09:50.287415 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:09:50.287626 392 external_mini_cluster.cc:1428] Setting key bbd61ea6023d1939df14621595426636
I20260924 07:09:50.287766 3149 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45911
I20260924 07:09:50.287870 3149 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:50.288106 3149 heartbeater.cc:511] Master 127.0.98.62:45911 requested a full tablet report, sending...
I20260924 07:09:50.288502 2824 ts_manager.cc:194] Registered new tserver with Master: ca3c698d8d4743ed95801d9714a751d7 (127.0.98.2:39777)
I20260924 07:09:50.288944 2824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55179
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 07:09:50.311617 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45911
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:50.390637 3154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:50.390827 3154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:50.390851 3154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:50.391989 3154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:50.392030 3154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:50.392057 3154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:09:50.393586 3154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:50.393875 3154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:50.394111 3154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:50.399072 3162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:50.399087 3159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:50.399123 3154 server_base.cc:1034] running on GCE node
W20260924 07:09:50.399087 3160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:50.399537 3154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:50.399770 3154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:50.400913 3154 hybrid_clock.cc:648] HybridClock initialized: now 1790233790400892 us; error 39 us; skew 500 ppm
Sep 24 07:09:50 dist-test-slave-14ww krb5kdc[2779](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:50.403355 3154 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:09:50.403789 3154 webserver.cc:460] Webserver started at http://127.0.98.3:38585/ using document root <none> and password file <none>
I20260924 07:09:50.403964 3154 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:50.404003 3154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:50.404088 3154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:50.405085 3154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-2/data/instance:
uuid: "73954029d7384972a3a80a314da04ebf"
format_stamp: "Formatted at 2026-09-24 07:09:50 on dist-test-slave-14ww"
server_key: "377403d400ae8da59657acc26d026e32"
server_key_iv: "897cbbec6dda4af5a527ccff16434933"
server_key_version: "encryptionkey@0"
I20260924 07:09:50.405457 3154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-2/wal/instance:
uuid: "73954029d7384972a3a80a314da04ebf"
format_stamp: "Formatted at 2026-09-24 07:09:50 on dist-test-slave-14ww"
server_key: "377403d400ae8da59657acc26d026e32"
server_key_iv: "897cbbec6dda4af5a527ccff16434933"
server_key_version: "encryptionkey@0"
I20260924 07:09:50.406692 3154 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:09:50.407382 3169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:50.407620 3154 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:09:50.407703 3154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-2/wal
uuid: "73954029d7384972a3a80a314da04ebf"
format_stamp: "Formatted at 2026-09-24 07:09:50 on dist-test-slave-14ww"
server_key: "377403d400ae8da59657acc26d026e32"
server_key_iv: "897cbbec6dda4af5a527ccff16434933"
server_key_version: "encryptionkey@0"
I20260924 07:09:50.407753 3154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:50.441041 3154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:50.442808 3154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:50.442921 3154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:50.443102 3154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:50.443410 3154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:50.443449 3154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:50.443470 3154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:50.443521 3154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:50.449124 3154 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42145
I20260924 07:09:50.449179 3282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42145 every 8 connection(s)
I20260924 07:09:50.449465 3154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 07:09:50 dist-test-slave-14ww krb5kdc[2779](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233790, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:09:50.456385 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 3154
I20260924 07:09:50.456521 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:09:50.456708 392 external_mini_cluster.cc:1428] Setting key 1d5e29fe2a84a78fbc7d86e847284418
I20260924 07:09:50.459334 2863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:50.450269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47837 (local address 127.0.98.62:45911)
0924 07:09:50.450335 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:50.450338 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:50.450589 (+ 251us) server_negotiation.cc:407] Connection header received
0924 07:09:50.451466 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:50.451467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:50.451477 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:50.451562 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:50.452143 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.452871 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.453935 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.454204 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.454303 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:50.456501 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:50.456502 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:50.456503 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:50.456511 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:50.457974 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:50.458402 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:50.458403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:50.458404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:50.458460 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:50.458710 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:50.458711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:50.458712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:50.458827 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:50.458903 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:50.459168 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 07:09:50.459243 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 07:09:50.459522 3285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:50.450266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45911 (local address 127.0.98.3:47837)
0924 07:09:50.450457 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:50.450493 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:50.451378 (+ 885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:50.451613 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:50.451616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:50.451851 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:50.452033 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:50.452037 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.452957 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.452958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:50.453840 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:50.453842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:50.454311 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:50.454311 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:50.454415 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:50.454937 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:50.454945 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:50.456404 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:50.458080 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:50.458086 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:50.458088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:50.458301 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:50.458539 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:50.458539 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:50.458540 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:50.458643 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:50.458894 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:50.458899 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:50.459040 (+ 141us) client_negotiation.cc:769] Sending connection context
0924 07:09:50.459147 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 07:09:50.459297 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":99,"threads_started":1}
I20260924 07:09:50.459932 3283 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45911
I20260924 07:09:50.460016 3283 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:50.460211 3283 heartbeater.cc:511] Master 127.0.98.62:45911 requested a full tablet report, sending...
I20260924 07:09:50.460588 2824 ts_manager.cc:194] Registered new tserver with Master: 73954029d7384972a3a80a314da04ebf (127.0.98.3:42145)
I20260924 07:09:50.460999 2824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47837
I20260924 07:09:50.468290 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 07:09:50.479398 2863 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:09:50.478517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39648 (local address 127.0.98.62:45911)
0924 07:09:50.478581 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:50.478584 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:50.478724 (+ 140us) server_negotiation.cc:407] Connection header received
0924 07:09:50.478766 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:50.478767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:50.478775 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:50.478824 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:50.479328 (+ 504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39648: BlockingRecv error: recv got EOF from 127.0.0.1:39648 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 07:09:50.480198 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 2886
I20260924 07:09:50.484478 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 3020
I20260924 07:09:50.488386 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 3154
I20260924 07:09:50.492612 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 2792
2026-09-24T07:09:50Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1834 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:09:50 dist-test-slave-14ww krb5kdc[3295](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:09:50 dist-test-slave-14ww krb5kdc[3295](info): set up 2 sockets
Sep 24 07:09:50 dist-test-slave-14ww krb5kdc[3295](info): commencing operation
krb5kdc: starting...
W20260924 07:09:51.532572 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:09:51 dist-test-slave-14ww krb5kdc[3295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:09:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:09:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:09:51.644286 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42469
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:51.722167 3308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:51.722398 3308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:51.722432 3308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:51.723564 3308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:09:51.723631 3308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:51.723658 3308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:51.723687 3308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:09:51.723711 3308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:09:51.725244 3308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:51.725505 3308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:51.725728 3308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:51.730340 3316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:51.730474 3313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:51.730536 3314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:51.730569 3308 server_base.cc:1034] running on GCE node
I20260924 07:09:51.730787 3308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:51.731062 3308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:51.732225 3308 hybrid_clock.cc:648] HybridClock initialized: now 1790233791732209 us; error 27 us; skew 500 ppm
Sep 24 07:09:51 dist-test-slave-14ww krb5kdc[3295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:51.734597 3308 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:09:51.735019 3308 webserver.cc:460] Webserver started at http://127.0.98.62:38463/ using document root <none> and password file <none>
I20260924 07:09:51.735226 3308 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:51.735268 3308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:51.735352 3308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:51.736302 3308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "63597e32efc54ddea35b98921b59df54"
format_stamp: "Formatted at 2026-09-24 07:09:51 on dist-test-slave-14ww"
server_key: "835fcb7d0a4e4057dcfb4deea88ac89f"
server_key_iv: "7b6026dfc66e4ce385f6ac3ae79ba549"
server_key_version: "encryptionkey@0"
I20260924 07:09:51.736579 3308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "63597e32efc54ddea35b98921b59df54"
format_stamp: "Formatted at 2026-09-24 07:09:51 on dist-test-slave-14ww"
server_key: "835fcb7d0a4e4057dcfb4deea88ac89f"
server_key_iv: "7b6026dfc66e4ce385f6ac3ae79ba549"
server_key_version: "encryptionkey@0"
I20260924 07:09:51.737831 3308 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 07:09:51.738493 3323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:51.738655 3308 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:09:51.738737 3308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "63597e32efc54ddea35b98921b59df54"
format_stamp: "Formatted at 2026-09-24 07:09:51 on dist-test-slave-14ww"
server_key: "835fcb7d0a4e4057dcfb4deea88ac89f"
server_key_iv: "7b6026dfc66e4ce385f6ac3ae79ba549"
server_key_version: "encryptionkey@0"
I20260924 07:09:51.738788 3308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:51.755725 3308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:51.757469 3308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:51.757572 3308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:51.761181 3308 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42469
I20260924 07:09:51.761247 3375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42469 every 8 connection(s)
I20260924 07:09:51.761547 3308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:09:51.762360 3376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:51.764460 3376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63597e32efc54ddea35b98921b59df54: Bootstrap starting.
I20260924 07:09:51.765072 3376 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63597e32efc54ddea35b98921b59df54: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:51.765308 3376 log.cc:824] T 00000000000000000000000000000000 P 63597e32efc54ddea35b98921b59df54: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:51.765908 3376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63597e32efc54ddea35b98921b59df54: No bootstrap required, opened a new log
I20260924 07:09:51.767382 3376 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63597e32efc54ddea35b98921b59df54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63597e32efc54ddea35b98921b59df54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42469 } }
I20260924 07:09:51.767511 3376 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63597e32efc54ddea35b98921b59df54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:51.767534 3376 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63597e32efc54ddea35b98921b59df54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63597e32efc54ddea35b98921b59df54, State: Initialized, Role: FOLLOWER
I20260924 07:09:51.767629 3376 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63597e32efc54ddea35b98921b59df54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63597e32efc54ddea35b98921b59df54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42469 } }
I20260924 07:09:51.767675 3376 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63597e32efc54ddea35b98921b59df54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:09:51.767695 3376 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63597e32efc54ddea35b98921b59df54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:09:51.767727 3376 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63597e32efc54ddea35b98921b59df54 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:51.768442 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 3308
I20260924 07:09:51.768458 3376 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63597e32efc54ddea35b98921b59df54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63597e32efc54ddea35b98921b59df54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42469 } }
I20260924 07:09:51.768534 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:09:51.768570 3376 leader_election.cc:302] T 00000000000000000000000000000000 P 63597e32efc54ddea35b98921b59df54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63597e32efc54ddea35b98921b59df54; no voters:
I20260924 07:09:51.768780 392 external_mini_cluster.cc:1428] Setting key a975e15720646a7df6d167c482a0e2b5
I20260924 07:09:51.768816 3376 leader_election.cc:288] T 00000000000000000000000000000000 P 63597e32efc54ddea35b98921b59df54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:09:51.768846 3379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63597e32efc54ddea35b98921b59df54 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:09:51.768966 3379 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63597e32efc54ddea35b98921b59df54 [term 1 LEADER]: Becoming Leader. State: Replica: 63597e32efc54ddea35b98921b59df54, State: Running, Role: LEADER
I20260924 07:09:51.769037 3379 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63597e32efc54ddea35b98921b59df54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63597e32efc54ddea35b98921b59df54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42469 } }
I20260924 07:09:51.769076 3376 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63597e32efc54ddea35b98921b59df54 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:09:51.769351 3379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63597e32efc54ddea35b98921b59df54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63597e32efc54ddea35b98921b59df54. Latest consensus state: current_term: 1 leader_uuid: "63597e32efc54ddea35b98921b59df54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63597e32efc54ddea35b98921b59df54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42469 } } }
I20260924 07:09:51.769419 3379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63597e32efc54ddea35b98921b59df54 [sys.catalog]: This master's current role is: LEADER
I20260924 07:09:51.769361 3381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63597e32efc54ddea35b98921b59df54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63597e32efc54ddea35b98921b59df54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63597e32efc54ddea35b98921b59df54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42469 } } }
I20260924 07:09:51.769462 3381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63597e32efc54ddea35b98921b59df54 [sys.catalog]: This master's current role is: LEADER
I20260924 07:09:51.769686 3386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:09:51.770207 3386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:09:51.772847 3386 catalog_manager.cc:1347] Generated new cluster ID: 0d1aa56abd3840e0812905a202265b86
I20260924 07:09:51.772898 3386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 07:09:51 dist-test-slave-14ww krb5kdc[3295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:09:51.780704 3384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:51.769364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47836 (local address 127.0.98.62:42469)
0924 07:09:51.769837 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:51.769847 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:51.769987 (+ 140us) server_negotiation.cc:407] Connection header received
0924 07:09:51.770118 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:51.770126 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:51.770185 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:51.771064 (+ 879us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:51.771175 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:51.772221 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:51.773135 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:51.773752 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:51.773880 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:51.776552 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:51.776556 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:51.776556 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:51.776567 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:51.779001 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:51.779423 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:51.779426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:51.779426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:51.779519 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:51.779736 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:51.779737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:51.779737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:51.780005 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:51.780156 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:51.780271 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 07:09:51.780493 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":61,"threads_started":1}
I20260924 07:09:51.781350 3386 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:09:51.781546 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:09:51.781978 3386 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:09:51.787322 3386 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63597e32efc54ddea35b98921b59df54: Generated new TSK 0
I20260924 07:09:51.787482 3386 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:09:51.795688 3384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:51.792135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47842 (local address 127.0.98.62:42469)
0924 07:09:51.792192 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:51.792194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:51.792331 (+ 137us) server_negotiation.cc:407] Connection header received
0924 07:09:51.792395 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:51.792395 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:51.792404 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:51.792450 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:51.792870 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:51.793338 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:51.794403 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:51.794613 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:51.794663 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:51.795289 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:51.795290 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:51.795290 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:09:51.795446 (+ 156us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:51.795459 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:09:51.795464 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:51.795508 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:51.795586 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 07:09:51.795605 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 07:09:51.797971 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 3308
2026-09-24T07:09:51Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1303 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:09:51 dist-test-slave-14ww krb5kdc[3409](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:09:51 dist-test-slave-14ww krb5kdc[3409](info): set up 2 sockets
Sep 24 07:09:51 dist-test-slave-14ww krb5kdc[3409](info): commencing operation
krb5kdc: starting...
W20260924 07:09:52.844528 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:09:52 dist-test-slave-14ww krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:09:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:09:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:09:52.964407 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35559
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35559
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:53.043030 3422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:53.043231 3422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:53.043252 3422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:53.044317 3422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:09:53.044351 3422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:53.044365 3422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:53.044378 3422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:09:53.044390 3422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:09:53.045782 3422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:53.045959 3422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:53.046126 3422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:53.050854 3427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:53.050972 3422 server_base.cc:1034] running on GCE node
W20260924 07:09:53.050865 3428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:53.050915 3430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:53.051357 3422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:53.051646 3422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:53.052796 3422 hybrid_clock.cc:648] HybridClock initialized: now 1790233793052777 us; error 49 us; skew 500 ppm
Sep 24 07:09:53 dist-test-slave-14ww krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:53.055083 3422 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:09:53.055524 3422 webserver.cc:460] Webserver started at http://127.0.98.62:34145/ using document root <none> and password file <none>
I20260924 07:09:53.055747 3422 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:53.055814 3422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:53.055982 3422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:53.057099 3422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "411d60f749fb40e0a3ee0ad52c74222b"
format_stamp: "Formatted at 2026-09-24 07:09:53 on dist-test-slave-14ww"
server_key: "ce851bf79d333e7b46675a38e5ffb578"
server_key_iv: "139779855beec9add9ab8a9416bc63f4"
server_key_version: "encryptionkey@0"
I20260924 07:09:53.057452 3422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "411d60f749fb40e0a3ee0ad52c74222b"
format_stamp: "Formatted at 2026-09-24 07:09:53 on dist-test-slave-14ww"
server_key: "ce851bf79d333e7b46675a38e5ffb578"
server_key_iv: "139779855beec9add9ab8a9416bc63f4"
server_key_version: "encryptionkey@0"
I20260924 07:09:53.058732 3422 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:09:53.059430 3437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:53.059607 3422 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:09:53.059705 3422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "411d60f749fb40e0a3ee0ad52c74222b"
format_stamp: "Formatted at 2026-09-24 07:09:53 on dist-test-slave-14ww"
server_key: "ce851bf79d333e7b46675a38e5ffb578"
server_key_iv: "139779855beec9add9ab8a9416bc63f4"
server_key_version: "encryptionkey@0"
I20260924 07:09:53.059793 3422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:53.070106 3422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:53.071755 3422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:53.071911 3422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:53.075757 3422 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35559
I20260924 07:09:53.075786 3489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35559 every 8 connection(s)
I20260924 07:09:53.076100 3422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:09:53.076987 3490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:53.078369 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 3422
I20260924 07:09:53.078493 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:09:53.078677 392 external_mini_cluster.cc:1428] Setting key e4af31ddb71914516c4d7012cfd59f52
I20260924 07:09:53.079486 3490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 411d60f749fb40e0a3ee0ad52c74222b: Bootstrap starting.
I20260924 07:09:53.080211 3490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 411d60f749fb40e0a3ee0ad52c74222b: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:53.080499 3490 log.cc:824] T 00000000000000000000000000000000 P 411d60f749fb40e0a3ee0ad52c74222b: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:53.081393 3490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 411d60f749fb40e0a3ee0ad52c74222b: No bootstrap required, opened a new log
I20260924 07:09:53.083629 3490 raft_consensus.cc:348] T 00000000000000000000000000000000 P 411d60f749fb40e0a3ee0ad52c74222b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "411d60f749fb40e0a3ee0ad52c74222b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35559 } }
I20260924 07:09:53.083789 3490 raft_consensus.cc:374] T 00000000000000000000000000000000 P 411d60f749fb40e0a3ee0ad52c74222b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:53.083830 3490 raft_consensus.cc:729] T 00000000000000000000000000000000 P 411d60f749fb40e0a3ee0ad52c74222b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 411d60f749fb40e0a3ee0ad52c74222b, State: Initialized, Role: FOLLOWER
I20260924 07:09:53.084007 3490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 411d60f749fb40e0a3ee0ad52c74222b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "411d60f749fb40e0a3ee0ad52c74222b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35559 } }
I20260924 07:09:53.084103 3490 raft_consensus.cc:388] T 00000000000000000000000000000000 P 411d60f749fb40e0a3ee0ad52c74222b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:09:53.084127 3490 raft_consensus.cc:482] T 00000000000000000000000000000000 P 411d60f749fb40e0a3ee0ad52c74222b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:09:53.084218 3490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 411d60f749fb40e0a3ee0ad52c74222b [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:53.084753 3490 raft_consensus.cc:504] T 00000000000000000000000000000000 P 411d60f749fb40e0a3ee0ad52c74222b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "411d60f749fb40e0a3ee0ad52c74222b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35559 } }
I20260924 07:09:53.084880 3490 leader_election.cc:302] T 00000000000000000000000000000000 P 411d60f749fb40e0a3ee0ad52c74222b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 411d60f749fb40e0a3ee0ad52c74222b; no voters:
I20260924 07:09:53.085050 3490 leader_election.cc:288] T 00000000000000000000000000000000 P 411d60f749fb40e0a3ee0ad52c74222b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:09:53.085073 3495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 411d60f749fb40e0a3ee0ad52c74222b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:09:53.085181 3495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 411d60f749fb40e0a3ee0ad52c74222b [term 1 LEADER]: Becoming Leader. State: Replica: 411d60f749fb40e0a3ee0ad52c74222b, State: Running, Role: LEADER
I20260924 07:09:53.085304 3490 sys_catalog.cc:564] T 00000000000000000000000000000000 P 411d60f749fb40e0a3ee0ad52c74222b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:09:53.085310 3495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 411d60f749fb40e0a3ee0ad52c74222b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "411d60f749fb40e0a3ee0ad52c74222b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35559 } }
I20260924 07:09:53.085660 3495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 411d60f749fb40e0a3ee0ad52c74222b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 411d60f749fb40e0a3ee0ad52c74222b. Latest consensus state: current_term: 1 leader_uuid: "411d60f749fb40e0a3ee0ad52c74222b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "411d60f749fb40e0a3ee0ad52c74222b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35559 } } }
I20260924 07:09:53.085733 3495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 411d60f749fb40e0a3ee0ad52c74222b [sys.catalog]: This master's current role is: LEADER
Sep 24 07:09:53 dist-test-slave-14ww krb5kdc[3409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:09:53.086365 3501 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:09:53.086432 3496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 411d60f749fb40e0a3ee0ad52c74222b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "411d60f749fb40e0a3ee0ad52c74222b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "411d60f749fb40e0a3ee0ad52c74222b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35559 } } }
I20260924 07:09:53.086525 3496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 411d60f749fb40e0a3ee0ad52c74222b [sys.catalog]: This master's current role is: LEADER
I20260924 07:09:53.087101 3501 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:09:53.088773 3501 catalog_manager.cc:1347] Generated new cluster ID: 50427910318e4f4c988cceba05601cda
I20260924 07:09:53.088822 3501 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:09:53.090579 3493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:53.079155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34520 (local address 127.0.98.62:35559)
0924 07:09:53.079340 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:53.079349 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:53.079380 (+ 31us) server_negotiation.cc:407] Connection header received
0924 07:09:53.079951 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:53.079959 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:53.080016 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:53.080166 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:53.080933 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.082380 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.083168 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.083789 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.083915 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:53.087029 (+ 3114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:53.087033 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:53.087033 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:53.087044 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:53.089072 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:53.089446 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:53.089449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:53.089450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:53.089561 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:53.089787 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:53.089788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:53.089788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:53.090019 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:53.090171 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:53.090284 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 07:09:53.090416 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":79,"threads_started":1}
I20260924 07:09:53.111450 3501 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:09:53.112021 3501 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:09:53.116808 3501 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 411d60f749fb40e0a3ee0ad52c74222b: Generated new TSK 0
I20260924 07:09:53.116982 3501 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 07:09:53.152316 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35559
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:53.245927 3516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:53.246184 3516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:53.246232 3516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:53.247992 3516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:53.248040 3516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:53.248075 3516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:09:53.249993 3516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:53.250216 3516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:53.250439 3516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:53.255367 3522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:53.255439 3521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:53.255374 3524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:53.255554 3516 server_base.cc:1034] running on GCE node
I20260924 07:09:53.255772 3516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:53.256012 3516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:53.257159 3516 hybrid_clock.cc:648] HybridClock initialized: now 1790233793257129 us; error 39 us; skew 500 ppm
Sep 24 07:09:53 dist-test-slave-14ww krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:53.259600 3516 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:09:53.260012 3516 webserver.cc:460] Webserver started at http://127.0.98.1:34041/ using document root <none> and password file <none>
I20260924 07:09:53.260188 3516 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:53.260238 3516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:53.260322 3516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:53.261291 3516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-0/data/instance:
uuid: "8bdd1bfc76b5487bb329bc11b9161b1a"
format_stamp: "Formatted at 2026-09-24 07:09:53 on dist-test-slave-14ww"
server_key: "8775ed382ca383fc40c90769c34eb7c7"
server_key_iv: "e3d1eae10e2abeb72e4ed187df0aebf4"
server_key_version: "encryptionkey@0"
I20260924 07:09:53.261581 3516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8bdd1bfc76b5487bb329bc11b9161b1a"
format_stamp: "Formatted at 2026-09-24 07:09:53 on dist-test-slave-14ww"
server_key: "8775ed382ca383fc40c90769c34eb7c7"
server_key_iv: "e3d1eae10e2abeb72e4ed187df0aebf4"
server_key_version: "encryptionkey@0"
I20260924 07:09:53.262784 3516 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:09:53.263522 3531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:53.263703 3516 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:09:53.263785 3516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-0/wal
uuid: "8bdd1bfc76b5487bb329bc11b9161b1a"
format_stamp: "Formatted at 2026-09-24 07:09:53 on dist-test-slave-14ww"
server_key: "8775ed382ca383fc40c90769c34eb7c7"
server_key_iv: "e3d1eae10e2abeb72e4ed187df0aebf4"
server_key_version: "encryptionkey@0"
I20260924 07:09:53.263836 3516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:53.277817 3516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:53.279503 3516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:53.279605 3516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:53.279758 3516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:53.279978 3516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:53.280006 3516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:53.280026 3516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:53.280038 3516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:53.285422 3516 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40893
I20260924 07:09:53.285498 3644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40893 every 8 connection(s)
I20260924 07:09:53.285817 3516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:09:53.287606 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 3516
I20260924 07:09:53.287688 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:09:53.287932 392 external_mini_cluster.cc:1428] Setting key ad5fc7120689a9d66ae32d43e9649ded
Sep 24 07:09:53 dist-test-slave-14ww krb5kdc[3409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 07:09:53.296329 3493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:53.286664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49117 (local address 127.0.98.62:35559)
0924 07:09:53.286829 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:53.286831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:53.286845 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:09:53.287812 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:53.287813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:53.287826 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:53.287888 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:53.288688 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.289485 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.290908 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.291119 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.291179 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:53.293504 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:53.293506 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:53.293506 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:53.293516 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:53.294959 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:53.295385 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:53.295386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:53.295387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:53.295446 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:53.295707 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:53.295707 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:53.295708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:53.295818 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:53.295894 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:53.296153 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 07:09:53.296236 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260924 07:09:53.296514 3647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:53.286598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35559 (local address 127.0.98.1:49117)
0924 07:09:53.286745 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:53.286784 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:53.287720 (+ 936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:53.287953 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:53.287957 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:53.288330 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:53.288582 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:53.288588 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.289582 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.289583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:53.290817 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:53.290820 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.291199 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.291200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:53.291343 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:53.291808 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:53.291818 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:53.293388 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:53.295060 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:53.295067 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:53.295069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:53.295289 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:53.295528 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:53.295529 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:53.295530 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:53.295637 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:53.295885 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:53.295891 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:53.296027 (+ 136us) client_negotiation.cc:769] Sending connection context
0924 07:09:53.296134 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 07:09:53.296279 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
I20260924 07:09:53.297049 3645 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35559
I20260924 07:09:53.297137 3645 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:53.297266 3645 heartbeater.cc:511] Master 127.0.98.62:35559 requested a full tablet report, sending...
I20260924 07:09:53.297796 3454 ts_manager.cc:194] Registered new tserver with Master: 8bdd1bfc76b5487bb329bc11b9161b1a (127.0.98.1:40893)
I20260924 07:09:53.298391 3454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49117
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 07:09:53.316377 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35559
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:53.393922 3650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:53.394111 3650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:53.394133 3650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:53.395267 3650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:53.395306 3650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:53.395331 3650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:09:53.396697 3650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:53.396945 3650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:53.397169 3650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:53.402068 3655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:53.402141 3656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:53.402086 3658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:53.402125 3650 server_base.cc:1034] running on GCE node
I20260924 07:09:53.402508 3650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:53.402740 3650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:53.403898 3650 hybrid_clock.cc:648] HybridClock initialized: now 1790233793403883 us; error 30 us; skew 500 ppm
Sep 24 07:09:53 dist-test-slave-14ww krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:53.406267 3650 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:09:53.406693 3650 webserver.cc:460] Webserver started at http://127.0.98.2:44981/ using document root <none> and password file <none>
I20260924 07:09:53.406908 3650 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:53.406980 3650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:53.407133 3650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:53.408087 3650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-1/data/instance:
uuid: "bda6b88e95f545c9bc83ef38ecdeced5"
format_stamp: "Formatted at 2026-09-24 07:09:53 on dist-test-slave-14ww"
server_key: "8300db131bae579e2fe04c14c90b001f"
server_key_iv: "43cd2d0ccf4daf25e8c36a4f0162075b"
server_key_version: "encryptionkey@0"
I20260924 07:09:53.408435 3650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bda6b88e95f545c9bc83ef38ecdeced5"
format_stamp: "Formatted at 2026-09-24 07:09:53 on dist-test-slave-14ww"
server_key: "8300db131bae579e2fe04c14c90b001f"
server_key_iv: "43cd2d0ccf4daf25e8c36a4f0162075b"
server_key_version: "encryptionkey@0"
I20260924 07:09:53.409808 3650 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:09:53.410508 3665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:53.410660 3650 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:09:53.410737 3650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-1/wal
uuid: "bda6b88e95f545c9bc83ef38ecdeced5"
format_stamp: "Formatted at 2026-09-24 07:09:53 on dist-test-slave-14ww"
server_key: "8300db131bae579e2fe04c14c90b001f"
server_key_iv: "43cd2d0ccf4daf25e8c36a4f0162075b"
server_key_version: "encryptionkey@0"
I20260924 07:09:53.410787 3650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:53.431710 3650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:53.433431 3650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:53.433531 3650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:53.433682 3650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:53.433910 3650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:53.433938 3650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:53.433959 3650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:53.433981 3650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:53.439581 3650 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44533
I20260924 07:09:53.439630 3778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44533 every 8 connection(s)
I20260924 07:09:53.439903 3650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:09:53.440948 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 3650
I20260924 07:09:53.441064 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:09:53.441255 392 external_mini_cluster.cc:1428] Setting key a92af13931847db405ca663ee3212a35
Sep 24 07:09:53 dist-test-slave-14ww krb5kdc[3409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 07:09:53.449898 3493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:53.440510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47731 (local address 127.0.98.62:35559)
0924 07:09:53.440551 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:53.440553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:53.440743 (+ 190us) server_negotiation.cc:407] Connection header received
0924 07:09:53.441432 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:53.441434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:53.441447 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:53.441504 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:53.442705 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.443236 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.444380 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.444569 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.444627 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:53.446877 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:53.446878 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:53.446878 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:53.446885 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:53.448493 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:53.448943 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:53.448944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:53.448944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:53.449007 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:53.449277 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:53.449278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:53.449278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:53.449390 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:53.449459 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:53.449734 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 07:09:53.449811 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 07:09:53.450085 3781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:53.440475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35559 (local address 127.0.98.2:47731)
0924 07:09:53.440617 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:53.440655 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:53.441335 (+ 680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:53.441574 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:53.441578 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:53.442347 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:53.442602 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:53.442607 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.443329 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.443330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:53.444303 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:53.444306 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.444652 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.444653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:53.444817 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:53.445268 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:53.445277 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:53.446782 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:53.448568 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:53.448574 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:53.448576 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:53.448856 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:53.449076 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:53.449077 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:53.449077 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:53.449217 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:53.449470 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:53.449476 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:53.449601 (+ 125us) client_negotiation.cc:769] Sending connection context
0924 07:09:53.449698 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 07:09:53.449840 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":83,"threads_started":1}
I20260924 07:09:53.450553 3779 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35559
I20260924 07:09:53.450639 3779 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:53.450816 3779 heartbeater.cc:511] Master 127.0.98.62:35559 requested a full tablet report, sending...
I20260924 07:09:53.451151 3454 ts_manager.cc:194] Registered new tserver with Master: bda6b88e95f545c9bc83ef38ecdeced5 (127.0.98.2:44533)
I20260924 07:09:53.451529 3454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47731
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 07:09:53.472937 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35559
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:53.551458 3784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:53.551645 3784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:53.551668 3784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:53.552753 3784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:53.552783 3784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:53.552809 3784 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:09:53.554100 3784 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:53.554262 3784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:53.554433 3784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:53.559232 3790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:53.559401 3784 server_base.cc:1034] running on GCE node
W20260924 07:09:53.559245 3789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:53.559269 3792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:53.559685 3784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:53.559906 3784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:53.561057 3784 hybrid_clock.cc:648] HybridClock initialized: now 1790233793561044 us; error 27 us; skew 500 ppm
Sep 24 07:09:53 dist-test-slave-14ww krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:53.563437 3784 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:09:53.563902 3784 webserver.cc:460] Webserver started at http://127.0.98.3:39765/ using document root <none> and password file <none>
I20260924 07:09:53.564114 3784 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:53.564178 3784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:53.564325 3784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:53.565361 3784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-2/data/instance:
uuid: "e7c4729e98554b9295a80fd1834914b4"
format_stamp: "Formatted at 2026-09-24 07:09:53 on dist-test-slave-14ww"
server_key: "9a84eee39325c30be9f3be7dff8b5467"
server_key_iv: "67d6e9399fe953c1037a467e8a418ac7"
server_key_version: "encryptionkey@0"
I20260924 07:09:53.565713 3784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e7c4729e98554b9295a80fd1834914b4"
format_stamp: "Formatted at 2026-09-24 07:09:53 on dist-test-slave-14ww"
server_key: "9a84eee39325c30be9f3be7dff8b5467"
server_key_iv: "67d6e9399fe953c1037a467e8a418ac7"
server_key_version: "encryptionkey@0"
I20260924 07:09:53.567025 3784 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:09:53.567878 3799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:53.568073 3784 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:09:53.568172 3784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-2/wal
uuid: "e7c4729e98554b9295a80fd1834914b4"
format_stamp: "Formatted at 2026-09-24 07:09:53 on dist-test-slave-14ww"
server_key: "9a84eee39325c30be9f3be7dff8b5467"
server_key_iv: "67d6e9399fe953c1037a467e8a418ac7"
server_key_version: "encryptionkey@0"
I20260924 07:09:53.568248 3784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:53.591094 3784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:53.593034 3784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:53.593359 3784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:53.593611 3784 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:53.593925 3784 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:53.593982 3784 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:53.594026 3784 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:53.594048 3784 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:53.600296 3784 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37867
I20260924 07:09:53.600363 3912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37867 every 8 connection(s)
I20260924 07:09:53.600698 3784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 07:09:53 dist-test-slave-14ww krb5kdc[3409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:09:53.607355 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 3784
I20260924 07:09:53.607468 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790233772407728-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:09:53.607707 392 external_mini_cluster.cc:1428] Setting key b0aec4c9b90fe921c3d99457d5a17e4d
I20260924 07:09:53.610507 3493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:53.601399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38983 (local address 127.0.98.62:35559)
0924 07:09:53.601441 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:53.601444 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:53.601702 (+ 258us) server_negotiation.cc:407] Connection header received
0924 07:09:53.602242 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:53.602243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:53.602256 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:53.602320 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:53.603257 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.603765 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.604791 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.604988 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.605070 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:53.607101 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:53.607103 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:53.607103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:53.607110 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:53.609125 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:53.609546 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:53.609547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:53.609548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:53.609610 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:53.609857 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:53.609858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:53.609858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:53.609995 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:53.610072 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:53.610336 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 07:09:53.610409 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 07:09:53.610719 3915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:53.601382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35559 (local address 127.0.98.3:38983)
0924 07:09:53.601543 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:53.601604 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:53.602137 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:53.602390 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:53.602394 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:53.602958 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:53.603153 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:53.603158 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.603861 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.603863 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:53.604670 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:53.604672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.605075 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.605076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:53.605206 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:53.605666 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:53.605674 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:53.607012 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:53.609217 (+ 2205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:53.609223 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:53.609226 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:53.609449 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:53.609678 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:53.609679 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:53.609679 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:53.609787 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:53.610064 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:53.610070 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:53.610209 (+ 139us) client_negotiation.cc:769] Sending connection context
0924 07:09:53.610318 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 07:09:53.610486 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":75,"threads_started":1}
I20260924 07:09:53.611136 3913 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35559
I20260924 07:09:53.611238 3913 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:53.611424 3913 heartbeater.cc:511] Master 127.0.98.62:35559 requested a full tablet report, sending...
I20260924 07:09:53.611775 3454 ts_manager.cc:194] Registered new tserver with Master: e7c4729e98554b9295a80fd1834914b4 (127.0.98.3:37867)
I20260924 07:09:53.612152 3454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38983
I20260924 07:09:53.619781 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 07:09:53 dist-test-slave-14ww krb5kdc[3409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233793, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 07:09:53 dist-test-slave-14ww krb5kdc[3409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233793, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:09:53.641860 3493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:53.634563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34524 (local address 127.0.98.62:35559)
0924 07:09:53.634646 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:53.634649 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:53.634747 (+ 98us) server_negotiation.cc:407] Connection header received
0924 07:09:53.634844 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:53.634845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:53.634856 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:53.634907 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:53.635487 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.635999 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.636768 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:53.636990 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:53.637070 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:53.639195 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:53.639197 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:53.639197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:53.639203 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:53.640398 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:53.640696 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:53.640696 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:53.640697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:53.640869 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:53.641129 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:53.641130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:53.641130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:53.641288 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:53.641379 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:53.641730 (+ 351us) server_negotiation.cc:299] Negotiation successful
0924 07:09:53.641796 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260924 07:09:53.642421 3454 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34524
I20260924 07:09:53.643075 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 3516
I20260924 07:09:53.647527 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 3650
I20260924 07:09:53.651988 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 3784
I20260924 07:09:53.656023 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 3422
2026-09-24T07:09:53Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1860 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:09:53 dist-test-slave-14ww krb5kdc[3925](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:09:53 dist-test-slave-14ww krb5kdc[3925](info): set up 2 sockets
Sep 24 07:09:53 dist-test-slave-14ww krb5kdc[3925](info): commencing operation
krb5kdc: starting...
W20260924 07:09:54.696612 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:09:54 dist-test-slave-14ww krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:09:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:09:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:09:54.809979 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40151
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40151
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:54.887749 3938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:54.887945 3938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:54.887969 3938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:54.889122 3938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:09:54.889158 3938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:54.889173 3938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:54.889184 3938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:09:54.889196 3938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:09:54.890595 3938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:54.890837 3938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:54.891064 3938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:54.895900 3943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:54.895860 3944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:54.895846 3946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:54.895874 3938 server_base.cc:1034] running on GCE node
I20260924 07:09:54.896250 3938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:54.896546 3938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:54.897729 3938 hybrid_clock.cc:648] HybridClock initialized: now 1790233794897707 us; error 50 us; skew 500 ppm
Sep 24 07:09:54 dist-test-slave-14ww krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:54.912196 3938 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:09:54.912701 3938 webserver.cc:460] Webserver started at http://127.0.98.62:46751/ using document root <none> and password file <none>
I20260924 07:09:54.912958 3938 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:54.913029 3938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:54.913183 3938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:54.914309 3938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "bb67174912014154aa436d9922b286af"
format_stamp: "Formatted at 2026-09-24 07:09:54 on dist-test-slave-14ww"
server_key: "1fbecb353207b6860975356a410c9a7d"
server_key_iv: "2f7ea04b16fc3e4e3978b828e3820781"
server_key_version: "encryptionkey@0"
I20260924 07:09:54.914683 3938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "bb67174912014154aa436d9922b286af"
format_stamp: "Formatted at 2026-09-24 07:09:54 on dist-test-slave-14ww"
server_key: "1fbecb353207b6860975356a410c9a7d"
server_key_iv: "2f7ea04b16fc3e4e3978b828e3820781"
server_key_version: "encryptionkey@0"
I20260924 07:09:54.916088 3938 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:09:54.916966 3953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:54.917140 3938 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:09:54.917246 3938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "bb67174912014154aa436d9922b286af"
format_stamp: "Formatted at 2026-09-24 07:09:54 on dist-test-slave-14ww"
server_key: "1fbecb353207b6860975356a410c9a7d"
server_key_iv: "2f7ea04b16fc3e4e3978b828e3820781"
server_key_version: "encryptionkey@0"
I20260924 07:09:54.917318 3938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:54.926740 3938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:54.928996 3938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:54.929150 3938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:54.932818 3938 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40151
I20260924 07:09:54.932832 4005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40151 every 8 connection(s)
I20260924 07:09:54.933192 3938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:09:54.933975 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 3938
I20260924 07:09:54.934002 4006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:54.934098 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:09:54.934293 392 external_mini_cluster.cc:1428] Setting key 3594e11f182d9cac235f1f406b26b057
I20260924 07:09:54.937356 4006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb67174912014154aa436d9922b286af: Bootstrap starting.
I20260924 07:09:54.938009 4006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb67174912014154aa436d9922b286af: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:54.938295 4006 log.cc:824] T 00000000000000000000000000000000 P bb67174912014154aa436d9922b286af: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:54.939004 4006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb67174912014154aa436d9922b286af: No bootstrap required, opened a new log
Sep 24 07:09:54 dist-test-slave-14ww krb5kdc[3925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:09:54.941007 4006 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb67174912014154aa436d9922b286af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb67174912014154aa436d9922b286af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40151 } }
I20260924 07:09:54.941190 4006 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb67174912014154aa436d9922b286af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:54.941236 4006 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb67174912014154aa436d9922b286af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb67174912014154aa436d9922b286af, State: Initialized, Role: FOLLOWER
I20260924 07:09:54.941370 4006 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb67174912014154aa436d9922b286af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb67174912014154aa436d9922b286af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40151 } }
I20260924 07:09:54.941460 4006 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb67174912014154aa436d9922b286af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:09:54.941491 4006 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb67174912014154aa436d9922b286af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:09:54.941532 4006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb67174912014154aa436d9922b286af [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:54.942076 4006 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb67174912014154aa436d9922b286af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb67174912014154aa436d9922b286af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40151 } }
I20260924 07:09:54.942195 4006 leader_election.cc:302] T 00000000000000000000000000000000 P bb67174912014154aa436d9922b286af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb67174912014154aa436d9922b286af; no voters:
I20260924 07:09:54.942373 4006 leader_election.cc:288] T 00000000000000000000000000000000 P bb67174912014154aa436d9922b286af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:09:54.942430 4011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb67174912014154aa436d9922b286af [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:09:54.942590 4011 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb67174912014154aa436d9922b286af [term 1 LEADER]: Becoming Leader. State: Replica: bb67174912014154aa436d9922b286af, State: Running, Role: LEADER
I20260924 07:09:54.942631 4006 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb67174912014154aa436d9922b286af [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:09:54.943014 4011 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb67174912014154aa436d9922b286af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb67174912014154aa436d9922b286af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40151 } }
I20260924 07:09:54.944483 4012 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb67174912014154aa436d9922b286af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb67174912014154aa436d9922b286af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb67174912014154aa436d9922b286af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40151 } } }
I20260924 07:09:54.944926 4012 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb67174912014154aa436d9922b286af [sys.catalog]: This master's current role is: LEADER
I20260924 07:09:54.944684 4013 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb67174912014154aa436d9922b286af [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb67174912014154aa436d9922b286af. Latest consensus state: current_term: 1 leader_uuid: "bb67174912014154aa436d9922b286af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb67174912014154aa436d9922b286af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40151 } } }
W20260924 07:09:54.944773 4026 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bb67174912014154aa436d9922b286af: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260924 07:09:54.945101 4013 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb67174912014154aa436d9922b286af [sys.catalog]: This master's current role is: LEADER
W20260924 07:09:54.945116 4026 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:09:54.945248 4028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:09:54.945452 4028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:09:54.945755 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:54.934812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51552 (local address 127.0.98.62:40151)
0924 07:09:54.934989 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:54.934999 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:54.935032 (+ 33us) server_negotiation.cc:407] Connection header received
0924 07:09:54.935584 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:54.935593 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:54.935673 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:54.935825 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:54.936379 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:54.937787 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:54.938588 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:54.939063 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:54.939193 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:54.941739 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:54.941743 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:54.941743 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:54.941753 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:54.943889 (+ 2136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:54.944233 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:54.944236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:54.944237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:54.944338 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:54.944567 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:54.944568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:54.944568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:54.944965 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:54.945130 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:54.945429 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 07:09:54.945572 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":76,"threads_started":1}
I20260924 07:09:54.947103 4028 catalog_manager.cc:1347] Generated new cluster ID: d39bd8abbdad4794862b58eded28bd53
I20260924 07:09:54.947152 4028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:09:54.986167 4028 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:09:54.986917 4028 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:09:54.994966 4028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb67174912014154aa436d9922b286af: Generated new TSK 0
I20260924 07:09:54.995137 4028 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 07:09:55.031746 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40151
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:55.117092 4032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:55.117295 4032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:55.117332 4032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:55.118465 4032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:55.118531 4032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:55.118577 4032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:09:55.120085 4032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:55.120321 4032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:55.120541 4032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:55.125298 4040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:55.125325 4038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:55.125303 4037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:55.125581 4032 server_base.cc:1034] running on GCE node
I20260924 07:09:55.125785 4032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:55.126006 4032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:55.127167 4032 hybrid_clock.cc:648] HybridClock initialized: now 1790233795127152 us; error 26 us; skew 500 ppm
Sep 24 07:09:55 dist-test-slave-14ww krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:55.129609 4032 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:09:55.129997 4032 webserver.cc:460] Webserver started at http://127.0.98.1:44391/ using document root <none> and password file <none>
I20260924 07:09:55.130214 4032 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:55.130283 4032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:55.130430 4032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:55.131397 4032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-0/data/instance:
uuid: "9b64744fbb64421aaa1e36b2e94f8d28"
format_stamp: "Formatted at 2026-09-24 07:09:55 on dist-test-slave-14ww"
server_key: "d159b060d39101f81aa2a461a0ad3913"
server_key_iv: "4397aa7a80624e7d33cc4800cd647903"
server_key_version: "encryptionkey@0"
I20260924 07:09:55.131748 4032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9b64744fbb64421aaa1e36b2e94f8d28"
format_stamp: "Formatted at 2026-09-24 07:09:55 on dist-test-slave-14ww"
server_key: "d159b060d39101f81aa2a461a0ad3913"
server_key_iv: "4397aa7a80624e7d33cc4800cd647903"
server_key_version: "encryptionkey@0"
I20260924 07:09:55.133064 4032 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:09:55.133735 4047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:55.133894 4032 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:55.133991 4032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-0/wal
uuid: "9b64744fbb64421aaa1e36b2e94f8d28"
format_stamp: "Formatted at 2026-09-24 07:09:55 on dist-test-slave-14ww"
server_key: "d159b060d39101f81aa2a461a0ad3913"
server_key_iv: "4397aa7a80624e7d33cc4800cd647903"
server_key_version: "encryptionkey@0"
I20260924 07:09:55.134070 4032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:55.146211 4032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:55.147861 4032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:55.148015 4032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:55.148231 4032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:55.148507 4032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:55.148561 4032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:55.148599 4032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:55.148627 4032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:55.154268 4032 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37633
I20260924 07:09:55.154317 4160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37633 every 8 connection(s)
I20260924 07:09:55.154609 4032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:09:55.156201 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 4032
I20260924 07:09:55.156296 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:09:55.156481 392 external_mini_cluster.cc:1428] Setting key fb739a4af9bb2bd230888e4b8a871339
Sep 24 07:09:55 dist-test-slave-14ww krb5kdc[3925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 07:09:55.164188 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:55.155316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54405 (local address 127.0.98.62:40151)
0924 07:09:55.155376 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:55.155379 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:55.155593 (+ 214us) server_negotiation.cc:407] Connection header received
0924 07:09:55.156552 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:55.156553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:55.156565 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:55.156623 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:55.157345 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:55.157849 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:55.158853 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:55.159088 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:55.159177 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:55.161348 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:55.161350 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:55.161350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:55.161358 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:55.162773 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:55.163182 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:55.163183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:55.163184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:55.163244 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:55.163488 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:55.163489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:55.163490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:55.163607 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:55.163679 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:55.163994 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 07:09:55.164098 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 07:09:55.164321 4163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:55.155304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40151 (local address 127.0.98.1:54405)
0924 07:09:55.155462 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:55.155501 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:55.156445 (+ 944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:55.156695 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:55.156698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:55.157054 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:55.157250 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:55.157254 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:55.157952 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:55.157953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:55.158760 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:55.158762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:55.159186 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:55.159188 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:55.159304 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:55.159826 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:55.159834 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:55.161254 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:55.162868 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:55.162875 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:55.162877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:55.163093 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:55.163312 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:55.163313 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:55.163314 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:55.163421 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:55.163685 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:55.163691 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:55.163850 (+ 159us) client_negotiation.cc:769] Sending connection context
0924 07:09:55.163963 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 07:09:55.164109 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":94,"threads_started":1}
I20260924 07:09:55.164871 4161 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40151
I20260924 07:09:55.164947 4161 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:55.165135 4161 heartbeater.cc:511] Master 127.0.98.62:40151 requested a full tablet report, sending...
I20260924 07:09:55.165618 3970 ts_manager.cc:194] Registered new tserver with Master: 9b64744fbb64421aaa1e36b2e94f8d28 (127.0.98.1:37633)
I20260924 07:09:55.166234 3970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54405
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 07:09:55.179888 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40151
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:55.259744 4166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:55.259927 4166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:55.259949 4166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:55.261104 4166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:55.261143 4166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:55.261168 4166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:09:55.262537 4166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:55.262720 4166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:55.262879 4166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:55.268090 4171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:55.268151 4174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:55.268231 4172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:55.268422 4166 server_base.cc:1034] running on GCE node
I20260924 07:09:55.268692 4166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:55.268946 4166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:55.270085 4166 hybrid_clock.cc:648] HybridClock initialized: now 1790233795270073 us; error 28 us; skew 500 ppm
Sep 24 07:09:55 dist-test-slave-14ww krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:55.272574 4166 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:09:55.273066 4166 webserver.cc:460] Webserver started at http://127.0.98.2:39455/ using document root <none> and password file <none>
I20260924 07:09:55.273304 4166 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:55.273372 4166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:55.273521 4166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:55.274658 4166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-1/data/instance:
uuid: "a82941e2efc64604b47d0832f548a70f"
format_stamp: "Formatted at 2026-09-24 07:09:55 on dist-test-slave-14ww"
server_key: "abf9827b584d1d1ad641117c9ab7d131"
server_key_iv: "50d64ec86e85b2199fa0b32b999116bb"
server_key_version: "encryptionkey@0"
I20260924 07:09:55.275009 4166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a82941e2efc64604b47d0832f548a70f"
format_stamp: "Formatted at 2026-09-24 07:09:55 on dist-test-slave-14ww"
server_key: "abf9827b584d1d1ad641117c9ab7d131"
server_key_iv: "50d64ec86e85b2199fa0b32b999116bb"
server_key_version: "encryptionkey@0"
I20260924 07:09:55.276425 4166 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:09:55.277431 4181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:55.277659 4166 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 07:09:55.277767 4166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-1/wal
uuid: "a82941e2efc64604b47d0832f548a70f"
format_stamp: "Formatted at 2026-09-24 07:09:55 on dist-test-slave-14ww"
server_key: "abf9827b584d1d1ad641117c9ab7d131"
server_key_iv: "50d64ec86e85b2199fa0b32b999116bb"
server_key_version: "encryptionkey@0"
I20260924 07:09:55.277843 4166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:55.300910 4166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:55.303258 4166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:55.303419 4166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:55.303659 4166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:55.303960 4166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:55.304013 4166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:55.304055 4166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:55.304082 4166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:55.309739 4166 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39667
I20260924 07:09:55.309784 4294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39667 every 8 connection(s)
I20260924 07:09:55.310082 4166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:09:55.314342 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 4166
I20260924 07:09:55.314460 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:09:55.314632 392 external_mini_cluster.cc:1428] Setting key 81d3a85172673730fc6b3b56b09dfb1b
Sep 24 07:09:55 dist-test-slave-14ww krb5kdc[3925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:09:55.320291 4297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:55.310743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40151 (local address 127.0.98.2:52909)
0924 07:09:55.310908 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:55.310941 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:55.311864 (+ 923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:55.312118 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:55.312121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:55.312359 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:55.312534 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:55.312538 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:55.313311 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:55.313312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:55.314105 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:55.314107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:55.314539 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:55.314540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:55.314670 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:55.315210 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:55.315236 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:55.316937 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:55.318908 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:55.318916 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:55.318918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:55.319152 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:55.319397 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:55.319398 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:55.319398 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:55.319497 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:55.319783 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:55.319787 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:55.319934 (+ 147us) client_negotiation.cc:769] Sending connection context
0924 07:09:55.320017 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 07:09:55.320142 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":80,"threads_started":1}
I20260924 07:09:55.320294 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:55.310773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52909 (local address 127.0.98.62:40151)
0924 07:09:55.310852 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:55.310854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:55.311023 (+ 169us) server_negotiation.cc:407] Connection header received
0924 07:09:55.311981 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:55.311982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:55.311996 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:55.312047 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:55.312641 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:55.313217 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:55.314208 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:55.314456 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:55.314520 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:55.317043 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:55.317044 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:55.317044 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:55.317051 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:55.318794 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:55.319230 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:55.319231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:55.319232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:55.319293 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:55.319565 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:55.319566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:55.319567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:55.319720 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:55.319796 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:55.320066 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 07:09:55.320157 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 07:09:55.320914 4295 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40151
I20260924 07:09:55.321018 4295 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:55.321205 4295 heartbeater.cc:511] Master 127.0.98.62:40151 requested a full tablet report, sending...
I20260924 07:09:55.321616 3970 ts_manager.cc:194] Registered new tserver with Master: a82941e2efc64604b47d0832f548a70f (127.0.98.2:39667)
I20260924 07:09:55.322115 3970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52909
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 07:09:55.340440 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40151
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:55.418916 4300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:55.419124 4300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:55.419147 4300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:55.420351 4300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:55.420392 4300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:55.420418 4300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:09:55.421860 4300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:55.422044 4300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:55.422206 4300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:55.427167 4308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:55.427213 4306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:55.427199 4305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:55.427311 4300 server_base.cc:1034] running on GCE node
I20260924 07:09:55.427608 4300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:55.427788 4300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:55.428916 4300 hybrid_clock.cc:648] HybridClock initialized: now 1790233795428872 us; error 66 us; skew 500 ppm
Sep 24 07:09:55 dist-test-slave-14ww krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:55.431181 4300 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:09:55.431594 4300 webserver.cc:460] Webserver started at http://127.0.98.3:40159/ using document root <none> and password file <none>
I20260924 07:09:55.431757 4300 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:55.431793 4300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:55.431874 4300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:55.432826 4300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-2/data/instance:
uuid: "600176fa335540fdbf70ec5118bb6d2e"
format_stamp: "Formatted at 2026-09-24 07:09:55 on dist-test-slave-14ww"
server_key: "6bc4da8ee775558f2f832e4b9c7553f3"
server_key_iv: "8ff30caeafba954ac3230458a243b847"
server_key_version: "encryptionkey@0"
I20260924 07:09:55.433110 4300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-2/wal/instance:
uuid: "600176fa335540fdbf70ec5118bb6d2e"
format_stamp: "Formatted at 2026-09-24 07:09:55 on dist-test-slave-14ww"
server_key: "6bc4da8ee775558f2f832e4b9c7553f3"
server_key_iv: "8ff30caeafba954ac3230458a243b847"
server_key_version: "encryptionkey@0"
I20260924 07:09:55.434288 4300 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:09:55.434989 4315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:55.435169 4300 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:09:55.435261 4300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-2/wal
uuid: "600176fa335540fdbf70ec5118bb6d2e"
format_stamp: "Formatted at 2026-09-24 07:09:55 on dist-test-slave-14ww"
server_key: "6bc4da8ee775558f2f832e4b9c7553f3"
server_key_iv: "8ff30caeafba954ac3230458a243b847"
server_key_version: "encryptionkey@0"
I20260924 07:09:55.435338 4300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:55.443044 4300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:55.445348 4300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:55.445495 4300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:55.445756 4300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:55.446024 4300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:55.446075 4300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:55.446105 4300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:55.446132 4300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:55.451642 4300 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34721
I20260924 07:09:55.451689 4428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34721 every 8 connection(s)
I20260924 07:09:55.451975 4300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:09:55.454761 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 4300
I20260924 07:09:55.454843 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790233772407728-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:09:55.455013 392 external_mini_cluster.cc:1428] Setting key 41eef0a4cd5f7fa505a90461b65f79d9
Sep 24 07:09:55 dist-test-slave-14ww krb5kdc[3925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:09:55.460613 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:55.452651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36597 (local address 127.0.98.62:40151)
0924 07:09:55.452690 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:55.452692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:55.452906 (+ 214us) server_negotiation.cc:407] Connection header received
0924 07:09:55.453854 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:55.453855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:55.453865 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:55.453917 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:55.454430 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:55.455029 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:55.456300 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:55.456481 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:55.456533 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:55.458389 (+ 1856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:55.458390 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:55.458391 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:55.458397 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:55.459542 (+ 1145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:55.459892 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:55.459893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:55.459893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:55.459940 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:55.460145 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:55.460146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:55.460146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:55.460237 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:55.460285 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:55.460496 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 07:09:55.460550 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 07:09:55.460855 4431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:55.452619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40151 (local address 127.0.98.3:36597)
0924 07:09:55.452804 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:55.452844 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:55.453766 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:55.453963 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:55.453966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:55.454178 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:55.454339 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:55.454342 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:55.455183 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:55.455184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:55.456234 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:55.456237 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:55.456550 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:55.456551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:55.456645 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:55.457080 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:55.457087 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:55.458300 (+ 1213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:55.459609 (+ 1309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:55.459614 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:55.459616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:55.459825 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:55.459998 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:55.459998 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:55.459999 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:55.460090 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:55.460290 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:55.460294 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:55.460392 (+ 98us) client_negotiation.cc:769] Sending connection context
0924 07:09:55.460470 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 07:09:55.460583 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":86,"threads_started":1}
I20260924 07:09:55.461265 4429 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40151
I20260924 07:09:55.461333 4429 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:55.461498 4429 heartbeater.cc:511] Master 127.0.98.62:40151 requested a full tablet report, sending...
I20260924 07:09:55.461843 3970 ts_manager.cc:194] Registered new tserver with Master: 600176fa335540fdbf70ec5118bb6d2e (127.0.98.3:34721)
I20260924 07:09:55.462226 3970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36597
I20260924 07:09:55.466678 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 07:09:55 dist-test-slave-14ww krb5kdc[3925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 07:09:55 dist-test-slave-14ww krb5kdc[3925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 07:09:55.488016 4436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:55.480756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38566 (local address 127.0.98.1:37633)
0924 07:09:55.480916 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:55.480921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:55.480932 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:09:55.480959 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:55.480961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:55.480982 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:55.481064 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:55.481675 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:55.482388 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:55.483111 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:55.483404 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:55.483480 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:55.485596 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:55.485602 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:55.485603 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:55.485608 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:55.486976 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:55.487283 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:55.487285 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:55.487285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:55.487341 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:55.487584 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:55.487585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:55.487585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:55.487711 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:55.487821 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:55.487885 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 07:09:55.487940 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":72,"threads_started":1}
Sep 24 07:09:55 dist-test-slave-14ww krb5kdc[3925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:09:55.495376 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:55.489201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51556 (local address 127.0.98.62:40151)
0924 07:09:55.489275 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:55.489277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:55.489285 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:09:55.489337 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:55.489338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:55.489346 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:55.489391 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:55.489822 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:55.490277 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:55.490885 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:55.491075 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:55.491128 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:55.493129 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:55.493131 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:55.493131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:55.493136 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:55.494238 (+ 1102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:55.494535 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:55.494536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:55.494536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:55.494579 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:55.494807 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:55.494808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:55.494808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:55.494959 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:55.495088 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:55.495209 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 07:09:55.495282 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260924 07:09:55.495596 3970 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51556
I20260924 07:09:55.495898 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 4032
I20260924 07:09:55.501138 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 4166
I20260924 07:09:55.505234 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 4300
I20260924 07:09:55.509219 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 3938
2026-09-24T07:09:55Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1853 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:09:55 dist-test-slave-14ww krb5kdc[4441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:09:55 dist-test-slave-14ww krb5kdc[4441](info): set up 2 sockets
Sep 24 07:09:55 dist-test-slave-14ww krb5kdc[4441](info): commencing operation
krb5kdc: starting...
W20260924 07:09:56.548548 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:09:56 dist-test-slave-14ww krb5kdc[4441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:09:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:09:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:09:56.667182 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42201
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:56.744846 4454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:56.745038 4454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:56.745060 4454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:56.746115 4454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:09:56.746147 4454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:56.746161 4454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:56.746174 4454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:09:56.746186 4454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:09:56.747545 4454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:56.747715 4454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:56.747869 4454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:56.752334 4460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:56.752336 4462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:56.752329 4459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:56.752487 4454 server_base.cc:1034] running on GCE node
I20260924 07:09:56.752884 4454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:56.753170 4454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:56.754338 4454 hybrid_clock.cc:648] HybridClock initialized: now 1790233796754323 us; error 53 us; skew 500 ppm
Sep 24 07:09:56 dist-test-slave-14ww krb5kdc[4441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:56.756910 4454 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:09:56.757265 4454 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:56.757336 4454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:56.757471 4454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:56.758509 4454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "dd35066f86f84b8ea0bf25ed97f83449"
format_stamp: "Formatted at 2026-09-24 07:09:56 on dist-test-slave-14ww"
server_key: "ec3c8f16c8b6fb6d67975de5c1feb00f"
server_key_iv: "7e336d0abceb6d48b5ace621e2071d5e"
server_key_version: "encryptionkey@0"
I20260924 07:09:56.758865 4454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "dd35066f86f84b8ea0bf25ed97f83449"
format_stamp: "Formatted at 2026-09-24 07:09:56 on dist-test-slave-14ww"
server_key: "ec3c8f16c8b6fb6d67975de5c1feb00f"
server_key_iv: "7e336d0abceb6d48b5ace621e2071d5e"
server_key_version: "encryptionkey@0"
I20260924 07:09:56.760138 4454 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:09:56.760926 4468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:56.761183 4454 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 07:09:56.761286 4454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "dd35066f86f84b8ea0bf25ed97f83449"
format_stamp: "Formatted at 2026-09-24 07:09:56 on dist-test-slave-14ww"
server_key: "ec3c8f16c8b6fb6d67975de5c1feb00f"
server_key_iv: "7e336d0abceb6d48b5ace621e2071d5e"
server_key_version: "encryptionkey@0"
I20260924 07:09:56.761371 4454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:56.781934 4454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:56.784216 4454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:56.784374 4454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:56.788297 4454 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42201
I20260924 07:09:56.788336 4520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42201 every 8 connection(s)
I20260924 07:09:56.788635 4454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:09:56.789499 4521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:56.791261 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 4454
I20260924 07:09:56.791352 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:09:56.791544 392 external_mini_cluster.cc:1428] Setting key c616a53ce29cd1474dbd77cfebd49a25
I20260924 07:09:56.791962 4521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd35066f86f84b8ea0bf25ed97f83449: Bootstrap starting.
I20260924 07:09:56.792639 4521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd35066f86f84b8ea0bf25ed97f83449: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:56.792984 4521 log.cc:824] T 00000000000000000000000000000000 P dd35066f86f84b8ea0bf25ed97f83449: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:56.793678 4521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd35066f86f84b8ea0bf25ed97f83449: No bootstrap required, opened a new log
I20260924 07:09:56.795804 4521 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd35066f86f84b8ea0bf25ed97f83449 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd35066f86f84b8ea0bf25ed97f83449" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42201 } }
I20260924 07:09:56.795990 4521 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd35066f86f84b8ea0bf25ed97f83449 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:56.796022 4521 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd35066f86f84b8ea0bf25ed97f83449 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd35066f86f84b8ea0bf25ed97f83449, State: Initialized, Role: FOLLOWER
I20260924 07:09:56.796131 4521 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd35066f86f84b8ea0bf25ed97f83449 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd35066f86f84b8ea0bf25ed97f83449" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42201 } }
I20260924 07:09:56.796201 4521 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd35066f86f84b8ea0bf25ed97f83449 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:09:56.796229 4521 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd35066f86f84b8ea0bf25ed97f83449 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:09:56.796271 4521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd35066f86f84b8ea0bf25ed97f83449 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:56.796972 4521 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd35066f86f84b8ea0bf25ed97f83449 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd35066f86f84b8ea0bf25ed97f83449" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42201 } }
I20260924 07:09:56.797089 4521 leader_election.cc:302] T 00000000000000000000000000000000 P dd35066f86f84b8ea0bf25ed97f83449 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd35066f86f84b8ea0bf25ed97f83449; no voters:
I20260924 07:09:56.797238 4521 leader_election.cc:288] T 00000000000000000000000000000000 P dd35066f86f84b8ea0bf25ed97f83449 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:09:56.797273 4526 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd35066f86f84b8ea0bf25ed97f83449 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:09:56.797405 4526 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd35066f86f84b8ea0bf25ed97f83449 [term 1 LEADER]: Becoming Leader. State: Replica: dd35066f86f84b8ea0bf25ed97f83449, State: Running, Role: LEADER
I20260924 07:09:56.797490 4521 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd35066f86f84b8ea0bf25ed97f83449 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:09:56.797513 4526 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd35066f86f84b8ea0bf25ed97f83449 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd35066f86f84b8ea0bf25ed97f83449" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42201 } }
I20260924 07:09:56.797861 4526 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd35066f86f84b8ea0bf25ed97f83449 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd35066f86f84b8ea0bf25ed97f83449. Latest consensus state: current_term: 1 leader_uuid: "dd35066f86f84b8ea0bf25ed97f83449" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd35066f86f84b8ea0bf25ed97f83449" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42201 } } }
I20260924 07:09:56.797922 4526 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd35066f86f84b8ea0bf25ed97f83449 [sys.catalog]: This master's current role is: LEADER
I20260924 07:09:56.797883 4527 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd35066f86f84b8ea0bf25ed97f83449 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd35066f86f84b8ea0bf25ed97f83449" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd35066f86f84b8ea0bf25ed97f83449" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42201 } } }
I20260924 07:09:56.797964 4527 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd35066f86f84b8ea0bf25ed97f83449 [sys.catalog]: This master's current role is: LEADER
I20260924 07:09:56.798345 4532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:09:56.798841 4532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 07:09:56 dist-test-slave-14ww krb5kdc[4441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:09:56.800414 4532 catalog_manager.cc:1347] Generated new cluster ID: cecc738de1ef44d4b2c24da439fd8cc3
I20260924 07:09:56.800462 4532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:09:56.802987 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:56.792046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58288 (local address 127.0.98.62:42201)
0924 07:09:56.792192 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:56.792200 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:56.792232 (+ 32us) server_negotiation.cc:407] Connection header received
0924 07:09:56.792874 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:56.792883 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:56.792948 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:56.793098 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:56.793812 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.795147 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.795872 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.796475 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.796589 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:56.799417 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:56.799421 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:56.799421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:56.799431 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:56.801466 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:56.801833 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:56.801836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:56.801837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:56.801930 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:56.802143 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:56.802144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:56.802144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:56.802379 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:56.802559 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:56.802655 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 07:09:56.802786 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":71,"threads_started":1}
I20260924 07:09:56.816804 4532 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:09:56.817320 4532 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:09:56.822021 4532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd35066f86f84b8ea0bf25ed97f83449: Generated new TSK 0
I20260924 07:09:56.822192 4532 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 07:09:56.851732 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42201
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:56.931986 4547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:56.932181 4547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:56.932205 4547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:56.933279 4547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:56.933313 4547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:56.933338 4547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:09:56.934705 4547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:56.934880 4547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:56.935036 4547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:56.939977 4555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:56.940063 4547 server_base.cc:1034] running on GCE node
W20260924 07:09:56.939977 4553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:56.939983 4552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:56.940452 4547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:56.940677 4547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:56.941838 4547 hybrid_clock.cc:648] HybridClock initialized: now 1790233796941841 us; error 53 us; skew 500 ppm
Sep 24 07:09:56 dist-test-slave-14ww krb5kdc[4441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:56.944265 4547 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:09:56.944608 4547 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:56.944656 4547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:56.944790 4547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:56.945829 4547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-0/data/instance:
uuid: "0c34805e0b794e338ed73ebe27ff6a80"
format_stamp: "Formatted at 2026-09-24 07:09:56 on dist-test-slave-14ww"
server_key: "cd9c6a04d2378e1d6e295b5f54360b36"
server_key_iv: "548bb0ad6d294d08a13eac1dc687092a"
server_key_version: "encryptionkey@0"
I20260924 07:09:56.946122 4547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0c34805e0b794e338ed73ebe27ff6a80"
format_stamp: "Formatted at 2026-09-24 07:09:56 on dist-test-slave-14ww"
server_key: "cd9c6a04d2378e1d6e295b5f54360b36"
server_key_iv: "548bb0ad6d294d08a13eac1dc687092a"
server_key_version: "encryptionkey@0"
I20260924 07:09:56.947268 4547 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 07:09:56.947885 4561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:56.948041 4547 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:09:56.948134 4547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-0/wal
uuid: "0c34805e0b794e338ed73ebe27ff6a80"
format_stamp: "Formatted at 2026-09-24 07:09:56 on dist-test-slave-14ww"
server_key: "cd9c6a04d2378e1d6e295b5f54360b36"
server_key_iv: "548bb0ad6d294d08a13eac1dc687092a"
server_key_version: "encryptionkey@0"
I20260924 07:09:56.948195 4547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:56.968472 4547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:56.970285 4547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:56.970435 4547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:56.970654 4547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:56.970954 4547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:56.971009 4547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:56.971048 4547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:56.971077 4547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:56.976686 4547 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39425
I20260924 07:09:56.976783 4674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39425 every 8 connection(s)
I20260924 07:09:56.977032 4547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 07:09:56 dist-test-slave-14ww krb5kdc[4441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:09:56.985520 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:56.977766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41081 (local address 127.0.98.62:42201)
0924 07:09:56.977832 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:56.977834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:56.978001 (+ 167us) server_negotiation.cc:407] Connection header received
0924 07:09:56.978463 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:56.978464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:56.978474 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:56.978529 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:56.979366 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.979858 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.980849 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.981031 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.981085 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:56.983005 (+ 1920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:56.983006 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:56.983007 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:56.983013 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:56.984388 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:56.984725 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:56.984726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:56.984726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:56.984811 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:56.985027 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:56.985027 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:56.985027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:56.985131 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:56.985181 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:56.985396 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 07:09:56.985457 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 07:09:56.985700 4677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:56.977735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42201 (local address 127.0.98.1:41081)
0924 07:09:56.977882 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:56.977922 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:56.978374 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:56.978579 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:56.978582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:56.978826 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:56.979294 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:56.979298 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.979931 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.979932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:56.980699 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:56.980700 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:56.981099 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:56.981099 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:56.981193 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:56.981629 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:56.981636 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:56.982924 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:56.984455 (+ 1531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:56.984459 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:56.984460 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:56.984663 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:56.984880 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:56.984881 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:56.984881 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:56.984971 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:56.985187 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:56.985191 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:56.985294 (+ 103us) client_negotiation.cc:769] Sending connection context
0924 07:09:56.985373 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 07:09:56.985479 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260924 07:09:56.986241 4675 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42201
I20260924 07:09:56.986253 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 4547
I20260924 07:09:56.986325 4675 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:56.986373 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:09:56.986469 4675 heartbeater.cc:511] Master 127.0.98.62:42201 requested a full tablet report, sending...
I20260924 07:09:56.986611 392 external_mini_cluster.cc:1428] Setting key e7b6402ef81da437440371757e1c211c
I20260924 07:09:56.987026 4485 ts_manager.cc:194] Registered new tserver with Master: 0c34805e0b794e338ed73ebe27ff6a80 (127.0.98.1:39425)
I20260924 07:09:56.987738 4485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41081
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 07:09:57.012334 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42201
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:57.091425 4680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:57.091612 4680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:57.091636 4680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:57.092797 4680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:57.092834 4680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:57.092861 4680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:09:57.094301 4680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:57.094533 4680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:57.094762 4680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:57.099573 4688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:57.099641 4685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:57.099661 4686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:57.099741 4680 server_base.cc:1034] running on GCE node
I20260924 07:09:57.100042 4680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:57.100271 4680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:57.101405 4680 hybrid_clock.cc:648] HybridClock initialized: now 1790233797101393 us; error 28 us; skew 500 ppm
Sep 24 07:09:57 dist-test-slave-14ww krb5kdc[4441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:57.103668 4680 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:09:57.104031 4680 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:57.104099 4680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:57.104231 4680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:57.105266 4680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-1/data/instance:
uuid: "589b350792b44a589710f9d95ed64e99"
format_stamp: "Formatted at 2026-09-24 07:09:57 on dist-test-slave-14ww"
server_key: "78596a83ba04655cd46d77afbb654dbe"
server_key_iv: "f66187aada1678ae27eb693cef3e9129"
server_key_version: "encryptionkey@0"
I20260924 07:09:57.105613 4680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-1/wal/instance:
uuid: "589b350792b44a589710f9d95ed64e99"
format_stamp: "Formatted at 2026-09-24 07:09:57 on dist-test-slave-14ww"
server_key: "78596a83ba04655cd46d77afbb654dbe"
server_key_iv: "f66187aada1678ae27eb693cef3e9129"
server_key_version: "encryptionkey@0"
I20260924 07:09:57.106828 4680 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:09:57.107513 4694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:57.107697 4680 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:09:57.107788 4680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-1/wal
uuid: "589b350792b44a589710f9d95ed64e99"
format_stamp: "Formatted at 2026-09-24 07:09:57 on dist-test-slave-14ww"
server_key: "78596a83ba04655cd46d77afbb654dbe"
server_key_iv: "f66187aada1678ae27eb693cef3e9129"
server_key_version: "encryptionkey@0"
I20260924 07:09:57.107872 4680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:57.134428 4680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:57.136224 4680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:57.136379 4680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:57.136590 4680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:57.136912 4680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:57.136966 4680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:57.137001 4680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:57.137029 4680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:57.142743 4680 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44881
I20260924 07:09:57.142781 4807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44881 every 8 connection(s)
I20260924 07:09:57.143056 4680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:09:57.147282 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 4680
I20260924 07:09:57.147403 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:09:57.147586 392 external_mini_cluster.cc:1428] Setting key 527340a9902e4f76fe475d85914f6794
Sep 24 07:09:57 dist-test-slave-14ww krb5kdc[4441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:09:57.153518 4810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.143728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42201 (local address 127.0.98.2:57895)
0924 07:09:57.143872 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:57.143922 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:57.144454 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:57.144716 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:57.144720 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:57.145017 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:57.145532 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:57.145537 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.146278 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.146279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:57.147162 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:57.147164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.147638 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.147639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:57.147805 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:57.148379 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:57.148396 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:57.150013 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:57.152163 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:57.152168 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:57.152170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:57.152367 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:57.152597 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:57.152598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:57.152598 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:57.152703 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:57.153067 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:57.153072 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:57.153183 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 07:09:57.153258 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 07:09:57.153369 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":59,"threads_started":1}
I20260924 07:09:57.153573 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.143781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57895 (local address 127.0.98.62:42201)
0924 07:09:57.143828 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:57.143831 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:57.144020 (+ 189us) server_negotiation.cc:407] Connection header received
0924 07:09:57.144568 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:57.144569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:57.144582 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:57.144648 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:57.145638 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.146178 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.147268 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.147550 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.147631 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:57.150104 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:57.150106 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:57.150107 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:57.150116 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:57.152082 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:57.152452 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:57.152454 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:57.152454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:57.152523 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:57.152816 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:57.152817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:57.152817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:57.152996 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:57.153082 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:57.153333 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 07:09:57.153424 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 07:09:57.154263 4808 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42201
I20260924 07:09:57.154373 4808 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:57.154536 4808 heartbeater.cc:511] Master 127.0.98.62:42201 requested a full tablet report, sending...
I20260924 07:09:57.154870 4485 ts_manager.cc:194] Registered new tserver with Master: 589b350792b44a589710f9d95ed64e99 (127.0.98.2:44881)
I20260924 07:09:57.155243 4485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57895
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 07:09:57.176024 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42201
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:09:57.255072 4813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:57.255311 4813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:57.255353 4813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:57.256577 4813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:57.256649 4813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:57.256693 4813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:09:57.258208 4813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:57.258482 4813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:57.258721 4813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:57.264119 4819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:57.264184 4818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:57.264410 4821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:57.264710 4813 server_base.cc:1034] running on GCE node
I20260924 07:09:57.264940 4813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:57.265177 4813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:57.266346 4813 hybrid_clock.cc:648] HybridClock initialized: now 1790233797266330 us; error 28 us; skew 500 ppm
Sep 24 07:09:57 dist-test-slave-14ww krb5kdc[4441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:09:57.268867 4813 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:09:57.269241 4813 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:57.269310 4813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:57.269443 4813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:57.270493 4813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-2/data/instance:
uuid: "02fd4d1bb7ba46f5ab1f7beab8e5e489"
format_stamp: "Formatted at 2026-09-24 07:09:57 on dist-test-slave-14ww"
server_key: "269c7a0081edc948b27e7979e0696324"
server_key_iv: "b37f57944dad6716120ad48c27982911"
server_key_version: "encryptionkey@0"
I20260924 07:09:57.270844 4813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-2/wal/instance:
uuid: "02fd4d1bb7ba46f5ab1f7beab8e5e489"
format_stamp: "Formatted at 2026-09-24 07:09:57 on dist-test-slave-14ww"
server_key: "269c7a0081edc948b27e7979e0696324"
server_key_iv: "b37f57944dad6716120ad48c27982911"
server_key_version: "encryptionkey@0"
I20260924 07:09:57.272079 4813 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 07:09:57.272799 4827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:57.272953 4813 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:09:57.273038 4813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-2/wal
uuid: "02fd4d1bb7ba46f5ab1f7beab8e5e489"
format_stamp: "Formatted at 2026-09-24 07:09:57 on dist-test-slave-14ww"
server_key: "269c7a0081edc948b27e7979e0696324"
server_key_iv: "b37f57944dad6716120ad48c27982911"
server_key_version: "encryptionkey@0"
I20260924 07:09:57.273092 4813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:57.284108 4813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:57.286386 4813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:57.286494 4813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:57.286687 4813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:57.286924 4813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:57.286953 4813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:57.286974 4813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:57.286988 4813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:57.292513 4813 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43653
I20260924 07:09:57.292591 4940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43653 every 8 connection(s)
I20260924 07:09:57.292868 4813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 07:09:57 dist-test-slave-14ww krb5kdc[4441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:09:57.300098 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 4813
I20260924 07:09:57.300205 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:09:57.300411 392 external_mini_cluster.cc:1428] Setting key 0cb6502aabc7e36298545353ca43490e
I20260924 07:09:57.301908 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.293553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41897 (local address 127.0.98.62:42201)
0924 07:09:57.293596 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:57.293599 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:57.293808 (+ 209us) server_negotiation.cc:407] Connection header received
0924 07:09:57.294276 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:57.294277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:57.294287 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:57.294345 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:57.295183 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.295696 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.296615 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.296894 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.296948 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:57.299329 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:57.299330 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:57.299330 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:57.299336 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:57.300662 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:57.301119 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:57.301119 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:57.301120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:57.301167 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:57.301422 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:57.301423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:57.301423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:57.301529 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:57.301583 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:57.301794 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 07:09:57.301848 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 07:09:57.302080 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.293524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42201 (local address 127.0.98.3:41897)
0924 07:09:57.293682 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:57.293722 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:57.294185 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:57.294394 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:57.294397 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:57.294633 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:57.295101 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:57.295106 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.295775 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.295776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:57.296542 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:57.296544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.296962 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.296963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:57.297060 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:57.297610 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:09:57.297618 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:57.299247 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:57.300717 (+ 1470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:57.300724 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:57.300726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:57.301043 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:57.301229 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:09:57.301230 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:09:57.301230 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:09:57.301359 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:09:57.301596 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:57.301600 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:57.301703 (+ 103us) client_negotiation.cc:769] Sending connection context
0924 07:09:57.301783 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 07:09:57.301894 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260924 07:09:57.302510 4941 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42201
I20260924 07:09:57.302608 4941 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:57.302755 4941 heartbeater.cc:511] Master 127.0.98.62:42201 requested a full tablet report, sending...
I20260924 07:09:57.303135 4485 ts_manager.cc:194] Registered new tserver with Master: 02fd4d1bb7ba46f5ab1f7beab8e5e489 (127.0.98.3:43653)
I20260924 07:09:57.303457 4485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41897
I20260924 07:09:57.312057 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:09:57.318528 4524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.313266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58296 (local address 127.0.98.62:42201)
0924 07:09:57.313324 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:57.313326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:57.313461 (+ 135us) server_negotiation.cc:407] Connection header received
0924 07:09:57.313497 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:57.313498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:57.313507 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:57.313544 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:57.314021 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.314451 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.315093 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.315260 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.315310 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:57.316439 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:57.316440 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:57.316441 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:09:57.316446 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:57.317637 (+ 1191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:57.317918 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:57.317919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:57.317919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:57.317962 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:09:57.318112 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:09:57.318112 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:09:57.318112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:09:57.318221 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:57.318291 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:57.318371 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 07:09:57.318431 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 07:09:57.320112 4485 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:09:57.321774 4485 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:09:57.327816 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.323919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44881 (local address 127.0.0.1:40452)
0924 07:09:57.324330 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:57.324341 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:57.324401 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:57.324570 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:57.324573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:57.324575 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:57.324632 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:57.324634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.325593 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.325594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:57.326731 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:57.326733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.327539 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.327540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:57.327614 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:57.327627 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 07:09:57.327695 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 07:09:57.327747 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":1,"thread_start_us":45,"threads_started":1}
I20260924 07:09:57.327833 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.324091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40452 (local address 127.0.98.2:44881)
0924 07:09:57.324279 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:57.324284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:57.324390 (+ 106us) server_negotiation.cc:407] Connection header received
0924 07:09:57.324475 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:57.324477 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:57.324494 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:57.324562 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:57.324718 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.325490 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.326826 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.327443 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.327537 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:57.327602 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:57.327716 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 07:09:57.327756 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":64,"threads_started":1}
I20260924 07:09:57.327929 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.323919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39425 (local address 127.0.0.1:48654)
0924 07:09:57.324224 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:57.324263 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:57.324332 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:57.324583 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:57.324584 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:57.324586 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:57.324652 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:57.324654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.325982 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.325983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:57.326831 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:57.326833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.327685 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.327686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:57.327747 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:57.327750 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:09:57.327801 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 07:09:57.327839 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":42,"threads_started":1}
I20260924 07:09:57.328188 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.324039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48654 (local address 127.0.98.1:39425)
0924 07:09:57.324353 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:57.324358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:57.324368 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:09:57.324408 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:57.324410 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:57.324426 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:57.324499 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:57.324754 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.325861 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.326901 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.327535 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.327602 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:57.327652 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:57.328039 (+ 387us) server_negotiation.cc:299] Negotiation successful
0924 07:09:57.328083 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":95,"threads_started":1}
I20260924 07:09:57.328487 4742 tablet_service.cc:1467] Processing CreateTablet for tablet 339c247c11b547c5be1b6b0287770e5f (DEFAULT_TABLE table=test-table [id=490a93d8f90a403383b726a0e5391666]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:57.328598 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.324093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43653 (local address 127.0.0.1:49482)
0924 07:09:57.324423 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:57.324432 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:57.324483 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:57.324772 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:57.324773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:57.324774 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:57.324806 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:57.324808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.326005 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.326005 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:57.327097 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:57.327099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.328380 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.328382 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:57.328443 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:57.328447 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:09:57.328497 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 07:09:57.328535 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":22,"thread_start_us":68,"threads_started":1}
I20260924 07:09:57.328642 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.324183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49482 (local address 127.0.98.3:43653)
0924 07:09:57.324337 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:57.324342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:57.324477 (+ 135us) server_negotiation.cc:407] Connection header received
0924 07:09:57.324606 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:57.324608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:57.324622 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:57.324693 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:57.324898 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.325911 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.327177 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.327860 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.328289 (+ 429us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:57.328348 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:57.328530 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 07:09:57.328565 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":71,"threads_started":1}
I20260924 07:09:57.328831 4742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 339c247c11b547c5be1b6b0287770e5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:57.328769 4609 tablet_service.cc:1467] Processing CreateTablet for tablet 339c247c11b547c5be1b6b0287770e5f (DEFAULT_TABLE table=test-table [id=490a93d8f90a403383b726a0e5391666]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:57.329023 4609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 339c247c11b547c5be1b6b0287770e5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:57.329332 4875 tablet_service.cc:1467] Processing CreateTablet for tablet 339c247c11b547c5be1b6b0287770e5f (DEFAULT_TABLE table=test-table [id=490a93d8f90a403383b726a0e5391666]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:57.329545 4875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 339c247c11b547c5be1b6b0287770e5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:57.331977 4958 tablet_bootstrap.cc:492] T 339c247c11b547c5be1b6b0287770e5f P 02fd4d1bb7ba46f5ab1f7beab8e5e489: Bootstrap starting.
I20260924 07:09:57.332095 4960 tablet_bootstrap.cc:492] T 339c247c11b547c5be1b6b0287770e5f P 0c34805e0b794e338ed73ebe27ff6a80: Bootstrap starting.
I20260924 07:09:57.332638 4958 tablet_bootstrap.cc:654] T 339c247c11b547c5be1b6b0287770e5f P 02fd4d1bb7ba46f5ab1f7beab8e5e489: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:57.332648 4960 tablet_bootstrap.cc:654] T 339c247c11b547c5be1b6b0287770e5f P 0c34805e0b794e338ed73ebe27ff6a80: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:57.333345 4960 log.cc:824] T 339c247c11b547c5be1b6b0287770e5f P 0c34805e0b794e338ed73ebe27ff6a80: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:57.333345 4958 log.cc:824] T 339c247c11b547c5be1b6b0287770e5f P 02fd4d1bb7ba46f5ab1f7beab8e5e489: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:57.333674 4959 tablet_bootstrap.cc:492] T 339c247c11b547c5be1b6b0287770e5f P 589b350792b44a589710f9d95ed64e99: Bootstrap starting.
I20260924 07:09:57.334085 4960 tablet_bootstrap.cc:492] T 339c247c11b547c5be1b6b0287770e5f P 0c34805e0b794e338ed73ebe27ff6a80: No bootstrap required, opened a new log
I20260924 07:09:57.334139 4960 ts_tablet_manager.cc:1397] T 339c247c11b547c5be1b6b0287770e5f P 0c34805e0b794e338ed73ebe27ff6a80: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:09:57.334260 4959 tablet_bootstrap.cc:654] T 339c247c11b547c5be1b6b0287770e5f P 589b350792b44a589710f9d95ed64e99: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:57.334435 4958 tablet_bootstrap.cc:492] T 339c247c11b547c5be1b6b0287770e5f P 02fd4d1bb7ba46f5ab1f7beab8e5e489: No bootstrap required, opened a new log
I20260924 07:09:57.334511 4958 ts_tablet_manager.cc:1397] T 339c247c11b547c5be1b6b0287770e5f P 02fd4d1bb7ba46f5ab1f7beab8e5e489: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 07:09:57.334836 4959 log.cc:824] T 339c247c11b547c5be1b6b0287770e5f P 589b350792b44a589710f9d95ed64e99: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:57.335549 4959 tablet_bootstrap.cc:492] T 339c247c11b547c5be1b6b0287770e5f P 589b350792b44a589710f9d95ed64e99: No bootstrap required, opened a new log
I20260924 07:09:57.335621 4959 ts_tablet_manager.cc:1397] T 339c247c11b547c5be1b6b0287770e5f P 589b350792b44a589710f9d95ed64e99: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:09:57.336150 4958 raft_consensus.cc:348] T 339c247c11b547c5be1b6b0287770e5f P 02fd4d1bb7ba46f5ab1f7beab8e5e489 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c34805e0b794e338ed73ebe27ff6a80" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39425 } } peers { permanent_uuid: "589b350792b44a589710f9d95ed64e99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44881 } } peers { permanent_uuid: "02fd4d1bb7ba46f5ab1f7beab8e5e489" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43653 } }
I20260924 07:09:57.336261 4958 raft_consensus.cc:374] T 339c247c11b547c5be1b6b0287770e5f P 02fd4d1bb7ba46f5ab1f7beab8e5e489 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:57.336134 4960 raft_consensus.cc:348] T 339c247c11b547c5be1b6b0287770e5f P 0c34805e0b794e338ed73ebe27ff6a80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c34805e0b794e338ed73ebe27ff6a80" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39425 } } peers { permanent_uuid: "589b350792b44a589710f9d95ed64e99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44881 } } peers { permanent_uuid: "02fd4d1bb7ba46f5ab1f7beab8e5e489" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43653 } }
I20260924 07:09:57.336282 4958 raft_consensus.cc:729] T 339c247c11b547c5be1b6b0287770e5f P 02fd4d1bb7ba46f5ab1f7beab8e5e489 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fd4d1bb7ba46f5ab1f7beab8e5e489, State: Initialized, Role: FOLLOWER
I20260924 07:09:57.336289 4960 raft_consensus.cc:374] T 339c247c11b547c5be1b6b0287770e5f P 0c34805e0b794e338ed73ebe27ff6a80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:57.336321 4960 raft_consensus.cc:729] T 339c247c11b547c5be1b6b0287770e5f P 0c34805e0b794e338ed73ebe27ff6a80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c34805e0b794e338ed73ebe27ff6a80, State: Initialized, Role: FOLLOWER
I20260924 07:09:57.336400 4958 consensus_queue.cc:260] T 339c247c11b547c5be1b6b0287770e5f P 02fd4d1bb7ba46f5ab1f7beab8e5e489 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c34805e0b794e338ed73ebe27ff6a80" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39425 } } peers { permanent_uuid: "589b350792b44a589710f9d95ed64e99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44881 } } peers { permanent_uuid: "02fd4d1bb7ba46f5ab1f7beab8e5e489" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43653 } }
I20260924 07:09:57.336441 4960 consensus_queue.cc:260] T 339c247c11b547c5be1b6b0287770e5f P 0c34805e0b794e338ed73ebe27ff6a80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c34805e0b794e338ed73ebe27ff6a80" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39425 } } peers { permanent_uuid: "589b350792b44a589710f9d95ed64e99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44881 } } peers { permanent_uuid: "02fd4d1bb7ba46f5ab1f7beab8e5e489" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43653 } }
I20260924 07:09:57.336673 4958 ts_tablet_manager.cc:1428] T 339c247c11b547c5be1b6b0287770e5f P 02fd4d1bb7ba46f5ab1f7beab8e5e489: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 07:09:57.336714 4941 heartbeater.cc:503] Master 127.0.98.62:42201 was elected leader, sending a full tablet report...
I20260924 07:09:57.336762 4960 ts_tablet_manager.cc:1428] T 339c247c11b547c5be1b6b0287770e5f P 0c34805e0b794e338ed73ebe27ff6a80: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 07:09:57.336870 4675 heartbeater.cc:503] Master 127.0.98.62:42201 was elected leader, sending a full tablet report...
I20260924 07:09:57.337613 4959 raft_consensus.cc:348] T 339c247c11b547c5be1b6b0287770e5f P 589b350792b44a589710f9d95ed64e99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c34805e0b794e338ed73ebe27ff6a80" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39425 } } peers { permanent_uuid: "589b350792b44a589710f9d95ed64e99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44881 } } peers { permanent_uuid: "02fd4d1bb7ba46f5ab1f7beab8e5e489" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43653 } }
I20260924 07:09:57.337782 4959 raft_consensus.cc:374] T 339c247c11b547c5be1b6b0287770e5f P 589b350792b44a589710f9d95ed64e99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:57.337824 4959 raft_consensus.cc:729] T 339c247c11b547c5be1b6b0287770e5f P 589b350792b44a589710f9d95ed64e99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 589b350792b44a589710f9d95ed64e99, State: Initialized, Role: FOLLOWER
I20260924 07:09:57.337922 4959 consensus_queue.cc:260] T 339c247c11b547c5be1b6b0287770e5f P 589b350792b44a589710f9d95ed64e99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c34805e0b794e338ed73ebe27ff6a80" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39425 } } peers { permanent_uuid: "589b350792b44a589710f9d95ed64e99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44881 } } peers { permanent_uuid: "02fd4d1bb7ba46f5ab1f7beab8e5e489" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43653 } }
I20260924 07:09:57.338124 4959 ts_tablet_manager.cc:1428] T 339c247c11b547c5be1b6b0287770e5f P 589b350792b44a589710f9d95ed64e99: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:09:57.338181 4808 heartbeater.cc:503] Master 127.0.98.62:42201 was elected leader, sending a full tablet report...
I20260924 07:09:57.384235 4964 raft_consensus.cc:482] T 339c247c11b547c5be1b6b0287770e5f P 02fd4d1bb7ba46f5ab1f7beab8e5e489 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:09:57.384349 4964 raft_consensus.cc:504] T 339c247c11b547c5be1b6b0287770e5f P 02fd4d1bb7ba46f5ab1f7beab8e5e489 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c34805e0b794e338ed73ebe27ff6a80" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39425 } } peers { permanent_uuid: "589b350792b44a589710f9d95ed64e99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44881 } } peers { permanent_uuid: "02fd4d1bb7ba46f5ab1f7beab8e5e489" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43653 } }
I20260924 07:09:57.384683 4964 leader_election.cc:288] T 339c247c11b547c5be1b6b0287770e5f P 02fd4d1bb7ba46f5ab1f7beab8e5e489 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c34805e0b794e338ed73ebe27ff6a80 (127.0.98.1:39425), 589b350792b44a589710f9d95ed64e99 (127.0.98.2:44881)
I20260924 07:09:57.386300 4965 raft_consensus.cc:482] T 339c247c11b547c5be1b6b0287770e5f P 0c34805e0b794e338ed73ebe27ff6a80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:09:57.386417 4965 raft_consensus.cc:504] T 339c247c11b547c5be1b6b0287770e5f P 0c34805e0b794e338ed73ebe27ff6a80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c34805e0b794e338ed73ebe27ff6a80" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39425 } } peers { permanent_uuid: "589b350792b44a589710f9d95ed64e99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44881 } } peers { permanent_uuid: "02fd4d1bb7ba46f5ab1f7beab8e5e489" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43653 } }
I20260924 07:09:57.386969 4965 leader_election.cc:288] T 339c247c11b547c5be1b6b0287770e5f P 0c34805e0b794e338ed73ebe27ff6a80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 589b350792b44a589710f9d95ed64e99 (127.0.98.2:44881), 02fd4d1bb7ba46f5ab1f7beab8e5e489 (127.0.98.3:43653)
I20260924 07:09:57.388098 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.384859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44881 (local address 127.0.98.3:48333)
0924 07:09:57.384941 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:57.384952 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:57.384997 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:57.385189 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:57.385190 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:57.385192 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:57.385265 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:57.385266 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.385904 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.385905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:57.386928 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:57.386931 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.387851 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.387852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:57.387940 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:57.387942 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 07:09:57.387986 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 07:09:57.388026 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":2}
I20260924 07:09:57.388098 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.385001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56927 (local address 127.0.98.1:39425)
0924 07:09:57.385046 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:57.385048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:57.385141 (+ 93us) server_negotiation.cc:407] Connection header received
0924 07:09:57.385216 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:57.385217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:57.385228 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:57.385280 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:57.385424 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.385878 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.387165 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.387738 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.387809 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:57.387819 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:57.388009 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 07:09:57.388040 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 07:09:57.388340 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.384899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39425 (local address 127.0.98.3:56927)
0924 07:09:57.385057 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:57.385066 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:57.385152 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:57.385284 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:57.385285 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:57.385286 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:57.385356 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:57.385357 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.385965 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.385966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:57.387071 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:57.387073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.387851 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.387852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:57.387926 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:57.387929 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:09:57.388204 (+ 275us) client_negotiation.cc:240] Negotiation successful
0924 07:09:57.388243 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":14,"thread_start_us":82,"threads_started":1}
I20260924 07:09:57.388653 4629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "339c247c11b547c5be1b6b0287770e5f" candidate_uuid: "02fd4d1bb7ba46f5ab1f7beab8e5e489" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c34805e0b794e338ed73ebe27ff6a80" is_pre_election: true
I20260924 07:09:57.388705 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.384950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48333 (local address 127.0.98.2:44881)
0924 07:09:57.384994 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:57.384997 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:57.385007 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:09:57.385100 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:57.385100 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:57.385109 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:57.385186 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:57.385336 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.385812 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.387233 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.387757 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.387812 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:57.387820 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:57.388591 (+ 771us) server_negotiation.cc:299] Negotiation successful
0924 07:09:57.388642 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 07:09:57.388764 4629 raft_consensus.cc:2454] T 339c247c11b547c5be1b6b0287770e5f P 0c34805e0b794e338ed73ebe27ff6a80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fd4d1bb7ba46f5ab1f7beab8e5e489 in term 0.
I20260924 07:09:57.388945 4762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "339c247c11b547c5be1b6b0287770e5f" candidate_uuid: "02fd4d1bb7ba46f5ab1f7beab8e5e489" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "589b350792b44a589710f9d95ed64e99" is_pre_election: true
I20260924 07:09:57.389065 4762 raft_consensus.cc:2454] T 339c247c11b547c5be1b6b0287770e5f P 589b350792b44a589710f9d95ed64e99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02fd4d1bb7ba46f5ab1f7beab8e5e489 in term 0.
I20260924 07:09:57.389209 4828 leader_election.cc:302] T 339c247c11b547c5be1b6b0287770e5f P 02fd4d1bb7ba46f5ab1f7beab8e5e489 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fd4d1bb7ba46f5ab1f7beab8e5e489, 0c34805e0b794e338ed73ebe27ff6a80; no voters:
I20260924 07:09:57.389333 4964 raft_consensus.cc:2781] T 339c247c11b547c5be1b6b0287770e5f P 02fd4d1bb7ba46f5ab1f7beab8e5e489 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:09:57.389400 4964 raft_consensus.cc:482] T 339c247c11b547c5be1b6b0287770e5f P 02fd4d1bb7ba46f5ab1f7beab8e5e489 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:09:57.389432 4964 raft_consensus.cc:3037] T 339c247c11b547c5be1b6b0287770e5f P 02fd4d1bb7ba46f5ab1f7beab8e5e489 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:57.390268 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.387026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41333 (local address 127.0.98.3:43653)
0924 07:09:57.387069 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:57.387071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:57.387146 (+ 75us) server_negotiation.cc:407] Connection header received
0924 07:09:57.387238 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:57.387239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:57.387247 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:57.387272 (+ 25us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:57.387445 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.388144 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.389426 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.389904 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.389951 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:57.389959 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:57.390173 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 07:09:57.390215 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 07:09:57.390307 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.386867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43653 (local address 127.0.98.1:41333)
0924 07:09:57.387090 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:57.387101 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:57.387170 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:57.387325 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:57.387326 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:57.387327 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:57.387378 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:57.387379 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.388228 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.388229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:57.389333 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:57.389335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.390019 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.390020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:57.390089 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:57.390092 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:09:57.390158 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 07:09:57.390203 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":73,"threads_started":1}
I20260924 07:09:57.390323 4964 raft_consensus.cc:504] T 339c247c11b547c5be1b6b0287770e5f P 02fd4d1bb7ba46f5ab1f7beab8e5e489 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c34805e0b794e338ed73ebe27ff6a80" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39425 } } peers { permanent_uuid: "589b350792b44a589710f9d95ed64e99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44881 } } peers { permanent_uuid: "02fd4d1bb7ba46f5ab1f7beab8e5e489" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43653 } }
I20260924 07:09:57.390460 4677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.386827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44881 (local address 127.0.98.1:47463)
0924 07:09:57.386965 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:57.386975 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:57.387011 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:57.387651 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:57.387652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:57.387653 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:09:57.387697 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:57.387698 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.388564 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.388564 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:57.389640 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:57.389642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.390222 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.390223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:57.390295 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:57.390298 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:09:57.390330 (+ 32us) client_negotiation.cc:240] Negotiation successful
0924 07:09:57.390377 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260924 07:09:57.390507 4964 leader_election.cc:288] T 339c247c11b547c5be1b6b0287770e5f P 02fd4d1bb7ba46f5ab1f7beab8e5e489 [CANDIDATE]: Term 1 election: Requested vote from peers 0c34805e0b794e338ed73ebe27ff6a80 (127.0.98.1:39425), 589b350792b44a589710f9d95ed64e99 (127.0.98.2:44881)
I20260924 07:09:57.390564 4762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "339c247c11b547c5be1b6b0287770e5f" candidate_uuid: "02fd4d1bb7ba46f5ab1f7beab8e5e489" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "589b350792b44a589710f9d95ed64e99"
I20260924 07:09:57.390580 4895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "339c247c11b547c5be1b6b0287770e5f" candidate_uuid: "0c34805e0b794e338ed73ebe27ff6a80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fd4d1bb7ba46f5ab1f7beab8e5e489" is_pre_election: true
I20260924 07:09:57.390643 4629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "339c247c11b547c5be1b6b0287770e5f" candidate_uuid: "02fd4d1bb7ba46f5ab1f7beab8e5e489" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c34805e0b794e338ed73ebe27ff6a80"
I20260924 07:09:57.390635 4762 raft_consensus.cc:3037] T 339c247c11b547c5be1b6b0287770e5f P 589b350792b44a589710f9d95ed64e99 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:57.390681 4895 raft_consensus.cc:2379] T 339c247c11b547c5be1b6b0287770e5f P 02fd4d1bb7ba46f5ab1f7beab8e5e489 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0c34805e0b794e338ed73ebe27ff6a80 in current term 1: Already voted for candidate 02fd4d1bb7ba46f5ab1f7beab8e5e489 in this term.
I20260924 07:09:57.390695 4629 raft_consensus.cc:3037] T 339c247c11b547c5be1b6b0287770e5f P 0c34805e0b794e338ed73ebe27ff6a80 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:57.391377 4629 raft_consensus.cc:2454] T 339c247c11b547c5be1b6b0287770e5f P 0c34805e0b794e338ed73ebe27ff6a80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fd4d1bb7ba46f5ab1f7beab8e5e489 in term 1.
I20260924 07:09:57.391446 4762 raft_consensus.cc:2454] T 339c247c11b547c5be1b6b0287770e5f P 589b350792b44a589710f9d95ed64e99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02fd4d1bb7ba46f5ab1f7beab8e5e489 in term 1.
I20260924 07:09:57.391551 4828 leader_election.cc:302] T 339c247c11b547c5be1b6b0287770e5f P 02fd4d1bb7ba46f5ab1f7beab8e5e489 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fd4d1bb7ba46f5ab1f7beab8e5e489, 0c34805e0b794e338ed73ebe27ff6a80; no voters:
I20260924 07:09:57.391685 4964 raft_consensus.cc:2781] T 339c247c11b547c5be1b6b0287770e5f P 02fd4d1bb7ba46f5ab1f7beab8e5e489 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:09:57.391696 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.387084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47463 (local address 127.0.98.2:44881)
0924 07:09:57.387512 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:57.387514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:57.387520 (+ 6us) server_negotiation.cc:407] Connection header received
0924 07:09:57.387539 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:57.387540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:57.387550 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:57.387753 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:09:57.387781 (+ 28us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.388461 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.389739 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.390108 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.391513 (+ 1405us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:57.391525 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:57.391545 (+ 20us) server_negotiation.cc:299] Negotiation successful
0924 07:09:57.391583 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":95,"threads_started":1}
I20260924 07:09:57.391835 4964 raft_consensus.cc:686] T 339c247c11b547c5be1b6b0287770e5f P 02fd4d1bb7ba46f5ab1f7beab8e5e489 [term 1 LEADER]: Becoming Leader. State: Replica: 02fd4d1bb7ba46f5ab1f7beab8e5e489, State: Running, Role: LEADER
I20260924 07:09:57.391856 4762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "339c247c11b547c5be1b6b0287770e5f" candidate_uuid: "0c34805e0b794e338ed73ebe27ff6a80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "589b350792b44a589710f9d95ed64e99" is_pre_election: true
I20260924 07:09:57.391933 4762 raft_consensus.cc:2379] T 339c247c11b547c5be1b6b0287770e5f P 589b350792b44a589710f9d95ed64e99 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0c34805e0b794e338ed73ebe27ff6a80 in current term 1: Already voted for candidate 02fd4d1bb7ba46f5ab1f7beab8e5e489 in this term.
I20260924 07:09:57.392032 4964 consensus_queue.cc:237] T 339c247c11b547c5be1b6b0287770e5f P 02fd4d1bb7ba46f5ab1f7beab8e5e489 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c34805e0b794e338ed73ebe27ff6a80" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39425 } } peers { permanent_uuid: "589b350792b44a589710f9d95ed64e99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44881 } } peers { permanent_uuid: "02fd4d1bb7ba46f5ab1f7beab8e5e489" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43653 } }
I20260924 07:09:57.392130 4565 leader_election.cc:302] T 339c247c11b547c5be1b6b0287770e5f P 0c34805e0b794e338ed73ebe27ff6a80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0c34805e0b794e338ed73ebe27ff6a80; no voters: 02fd4d1bb7ba46f5ab1f7beab8e5e489, 589b350792b44a589710f9d95ed64e99
I20260924 07:09:57.392280 4965 raft_consensus.cc:2726] T 339c247c11b547c5be1b6b0287770e5f P 0c34805e0b794e338ed73ebe27ff6a80 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 07:09:57.392899 4485 catalog_manager.cc:5495] T 339c247c11b547c5be1b6b0287770e5f P 02fd4d1bb7ba46f5ab1f7beab8e5e489 reported cstate change: term changed from 0 to 1, leader changed from <none> to 02fd4d1bb7ba46f5ab1f7beab8e5e489 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "02fd4d1bb7ba46f5ab1f7beab8e5e489" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c34805e0b794e338ed73ebe27ff6a80" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "589b350792b44a589710f9d95ed64e99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02fd4d1bb7ba46f5ab1f7beab8e5e489" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43653 } health_report { overall_health: HEALTHY } } }
W20260924 07:09:57.393540 4809 tablet.cc:2369] T 339c247c11b547c5be1b6b0287770e5f P 589b350792b44a589710f9d95ed64e99: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:09:57.399338 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.396890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49498 (local address 127.0.98.3:43653)
0924 07:09:57.396927 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:57.396928 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:57.396974 (+ 46us) server_negotiation.cc:407] Connection header received
0924 07:09:57.397055 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:57.397056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:57.397070 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:57.397132 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:09:57.397267 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.397770 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.398570 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:57.398799 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:57.398856 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:57.398978 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:09:57.399130 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:09:57.399219 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:57.399264 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 07:09:57.399280 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 07:09:57.400862 4762 raft_consensus.cc:1261] T 339c247c11b547c5be1b6b0287770e5f P 589b350792b44a589710f9d95ed64e99 [term 1 FOLLOWER]: Refusing update from remote peer 02fd4d1bb7ba46f5ab1f7beab8e5e489: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:09:57.400874 4629 raft_consensus.cc:1261] T 339c247c11b547c5be1b6b0287770e5f P 0c34805e0b794e338ed73ebe27ff6a80 [term 1 FOLLOWER]: Refusing update from remote peer 02fd4d1bb7ba46f5ab1f7beab8e5e489: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:09:57.401149 4970 consensus_queue.cc:1035] T 339c247c11b547c5be1b6b0287770e5f P 02fd4d1bb7ba46f5ab1f7beab8e5e489 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c34805e0b794e338ed73ebe27ff6a80" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:09:57.401274 4964 consensus_queue.cc:1035] T 339c247c11b547c5be1b6b0287770e5f P 02fd4d1bb7ba46f5ab1f7beab8e5e489 [LEADER]: Connected to new peer: Peer: permanent_uuid: "589b350792b44a589710f9d95ed64e99" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:09:57.404292 4975 mvcc.cc:205] Tried to move back new op lower bound from 7332797634151636992 to 7332797634118610944. Current Snapshot: MvccSnapshot[applied={T|T < 7332797634151636992}]
I20260924 07:09:57.405540 4485 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:09:57.405632 4485 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:09:57.406333 4485 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dd35066f86f84b8ea0bf25ed97f83449: Sending DeleteTablet for 3 replicas of tablet 339c247c11b547c5be1b6b0287770e5f
I20260924 07:09:57.406662 4609 tablet_service.cc:1514] Processing DeleteTablet for tablet 339c247c11b547c5be1b6b0287770e5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48654
I20260924 07:09:57.406705 4875 tablet_service.cc:1514] Processing DeleteTablet for tablet 339c247c11b547c5be1b6b0287770e5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49482
I20260924 07:09:57.406719 4742 tablet_service.cc:1514] Processing DeleteTablet for tablet 339c247c11b547c5be1b6b0287770e5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40452
I20260924 07:09:57.406926 4982 tablet_replica.cc:330] T 339c247c11b547c5be1b6b0287770e5f P 0c34805e0b794e338ed73ebe27ff6a80: stopping tablet replica
I20260924 07:09:57.407044 4982 raft_consensus.cc:2229] T 339c247c11b547c5be1b6b0287770e5f P 0c34805e0b794e338ed73ebe27ff6a80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:09:57.407089 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 4547
I20260924 07:09:57.408844 4983 tablet_replica.cc:330] T 339c247c11b547c5be1b6b0287770e5f P 589b350792b44a589710f9d95ed64e99: stopping tablet replica
I20260924 07:09:57.408982 4983 raft_consensus.cc:2229] T 339c247c11b547c5be1b6b0287770e5f P 589b350792b44a589710f9d95ed64e99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:09:57.409143 4983 raft_consensus.cc:2258] T 339c247c11b547c5be1b6b0287770e5f P 589b350792b44a589710f9d95ed64e99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:09:57.409623 4983 ts_tablet_manager.cc:1905] T 339c247c11b547c5be1b6b0287770e5f P 589b350792b44a589710f9d95ed64e99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:09:57.411293 4984 tablet_replica.cc:330] T 339c247c11b547c5be1b6b0287770e5f P 02fd4d1bb7ba46f5ab1f7beab8e5e489: stopping tablet replica
I20260924 07:09:57.411432 4984 raft_consensus.cc:2229] T 339c247c11b547c5be1b6b0287770e5f P 02fd4d1bb7ba46f5ab1f7beab8e5e489 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:09:57.411448 4983 ts_tablet_manager.cc:1918] T 339c247c11b547c5be1b6b0287770e5f P 589b350792b44a589710f9d95ed64e99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:09:57.411520 4983 log.cc:1196] T 339c247c11b547c5be1b6b0287770e5f P 589b350792b44a589710f9d95ed64e99: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-1/wal/wals/339c247c11b547c5be1b6b0287770e5f
I20260924 07:09:57.411664 4984 raft_consensus.cc:2258] T 339c247c11b547c5be1b6b0287770e5f P 02fd4d1bb7ba46f5ab1f7beab8e5e489 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:09:57.411796 4983 ts_tablet_manager.cc:1939] T 339c247c11b547c5be1b6b0287770e5f P 589b350792b44a589710f9d95ed64e99: Deleting consensus metadata
I20260924 07:09:57.412153 4984 ts_tablet_manager.cc:1905] T 339c247c11b547c5be1b6b0287770e5f P 02fd4d1bb7ba46f5ab1f7beab8e5e489: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:09:57.412160 4472 catalog_manager.cc:4841] TS 589b350792b44a589710f9d95ed64e99 (127.0.98.2:44881): tablet 339c247c11b547c5be1b6b0287770e5f (table test-table [id=490a93d8f90a403383b726a0e5391666]) successfully deleted
W20260924 07:09:57.413034 4469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39425 (error 108)
I20260924 07:09:57.413314 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 4680
I20260924 07:09:57.413914 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.413399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39425 (local address 127.0.0.1:48666)
0924 07:09:57.413844 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:57.413880 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39425: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427}
I20260924 07:09:57.414685 4984 ts_tablet_manager.cc:1918] T 339c247c11b547c5be1b6b0287770e5f P 02fd4d1bb7ba46f5ab1f7beab8e5e489: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:09:57.414805 4984 log.cc:1196] T 339c247c11b547c5be1b6b0287770e5f P 02fd4d1bb7ba46f5ab1f7beab8e5e489: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790233772407728-392-0/minicluster-data/ts-2/wal/wals/339c247c11b547c5be1b6b0287770e5f
W20260924 07:09:57.414952 4469 catalog_manager.cc:4584] TS 0c34805e0b794e338ed73ebe27ff6a80 (127.0.98.1:39425): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 339c247c11b547c5be1b6b0287770e5f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39425: connect: Connection refused (error 111)
I20260924 07:09:57.415181 4984 ts_tablet_manager.cc:1939] T 339c247c11b547c5be1b6b0287770e5f P 02fd4d1bb7ba46f5ab1f7beab8e5e489: Deleting consensus metadata
I20260924 07:09:57.415581 4469 catalog_manager.cc:4841] TS 02fd4d1bb7ba46f5ab1f7beab8e5e489 (127.0.98.3:43653): tablet 339c247c11b547c5be1b6b0287770e5f (table test-table [id=490a93d8f90a403383b726a0e5391666]) successfully deleted
I20260924 07:09:57.418291 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 4813
I20260924 07:09:57.423321 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 4454
2026-09-24T07:09:57Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1914 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T07:09:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:09:57Z Disabled control of system clock
I20260924 07:09:57.450344 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35741
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 07:09:57.528673 4989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:57.528930 4989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:57.528952 4989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:57.530126 4989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:09:57.530164 4989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:57.530177 4989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:57.530189 4989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:09:57.530201 4989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:09:57.531446 4989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35741
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:57.531625 4989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:57.531840 4989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:57.536396 4994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:57.536460 4989 server_base.cc:1034] running on GCE node
W20260924 07:09:57.536381 4997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:57.536381 4995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:57.536835 4989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:57.537137 4989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:57.538307 4989 hybrid_clock.cc:648] HybridClock initialized: now 1790233797538288 us; error 31 us; skew 500 ppm
I20260924 07:09:57.539526 4989 webserver.cc:460] Webserver started at http://127.0.98.62:32923/ using document root <none> and password file <none>
I20260924 07:09:57.539772 4989 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:57.539841 4989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:57.539971 4989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:57.541134 4989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "a448988e73cf47b6bd0e528b8b5d9bac"
format_stamp: "Formatted at 2026-09-24 07:09:57 on dist-test-slave-14ww"
server_key: "6fbe76784b4f2eae13d4a39a626b440f"
server_key_iv: "895cf0bfa285c6f43a4487dd132e26a3"
server_key_version: "encryptionkey@0"
I20260924 07:09:57.541522 4989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "a448988e73cf47b6bd0e528b8b5d9bac"
format_stamp: "Formatted at 2026-09-24 07:09:57 on dist-test-slave-14ww"
server_key: "6fbe76784b4f2eae13d4a39a626b440f"
server_key_iv: "895cf0bfa285c6f43a4487dd132e26a3"
server_key_version: "encryptionkey@0"
I20260924 07:09:57.542807 4989 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 07:09:57.543527 5003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:57.543685 4989 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:09:57.543766 4989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "a448988e73cf47b6bd0e528b8b5d9bac"
format_stamp: "Formatted at 2026-09-24 07:09:57 on dist-test-slave-14ww"
server_key: "6fbe76784b4f2eae13d4a39a626b440f"
server_key_iv: "895cf0bfa285c6f43a4487dd132e26a3"
server_key_version: "encryptionkey@0"
I20260924 07:09:57.543817 4989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:57.546962 4989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:57.547165 4989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:57.547253 4989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:57.550792 4989 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35741
I20260924 07:09:57.550819 5055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35741 every 8 connection(s)
I20260924 07:09:57.551116 4989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:09:57.551709 5056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:57.553926 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 4989
I20260924 07:09:57.554030 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:09:57.554214 392 external_mini_cluster.cc:1428] Setting key 45945c526165048439fe89b048416e25
I20260924 07:09:57.554629 5056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a448988e73cf47b6bd0e528b8b5d9bac: Bootstrap starting.
I20260924 07:09:57.555195 5056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a448988e73cf47b6bd0e528b8b5d9bac: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:57.555531 5056 log.cc:824] T 00000000000000000000000000000000 P a448988e73cf47b6bd0e528b8b5d9bac: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:57.556591 5059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.554681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38680 (local address 127.0.98.62:35741)
0924 07:09:57.554844 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:57.554854 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:57.554882 (+ 28us) server_negotiation.cc:407] Connection header received
0924 07:09:57.555248 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:57.555254 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:57.555320 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:57.555449 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:57.556145 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:57.556148 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:57.556149 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:09:57.556153 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:57.556195 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:09:57.556208 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:57.556306 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:57.556339 (+ 33us) server_negotiation.cc:299] Negotiation successful
0924 07:09:57.556419 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":72,"threads_started":1}
I20260924 07:09:57.556617 5056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a448988e73cf47b6bd0e528b8b5d9bac: No bootstrap required, opened a new log
I20260924 07:09:57.558102 5056 raft_consensus.cc:348] T 00000000000000000000000000000000 P a448988e73cf47b6bd0e528b8b5d9bac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a448988e73cf47b6bd0e528b8b5d9bac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35741 } }
I20260924 07:09:57.558223 5056 raft_consensus.cc:374] T 00000000000000000000000000000000 P a448988e73cf47b6bd0e528b8b5d9bac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:57.558243 5056 raft_consensus.cc:729] T 00000000000000000000000000000000 P a448988e73cf47b6bd0e528b8b5d9bac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a448988e73cf47b6bd0e528b8b5d9bac, State: Initialized, Role: FOLLOWER
I20260924 07:09:57.558372 5056 consensus_queue.cc:260] T 00000000000000000000000000000000 P a448988e73cf47b6bd0e528b8b5d9bac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a448988e73cf47b6bd0e528b8b5d9bac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35741 } }
I20260924 07:09:57.558460 5056 raft_consensus.cc:388] T 00000000000000000000000000000000 P a448988e73cf47b6bd0e528b8b5d9bac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:09:57.558502 5056 raft_consensus.cc:482] T 00000000000000000000000000000000 P a448988e73cf47b6bd0e528b8b5d9bac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:09:57.558552 5056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a448988e73cf47b6bd0e528b8b5d9bac [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:57.559291 5056 raft_consensus.cc:504] T 00000000000000000000000000000000 P a448988e73cf47b6bd0e528b8b5d9bac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a448988e73cf47b6bd0e528b8b5d9bac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35741 } }
I20260924 07:09:57.559397 5056 leader_election.cc:302] T 00000000000000000000000000000000 P a448988e73cf47b6bd0e528b8b5d9bac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a448988e73cf47b6bd0e528b8b5d9bac; no voters:
I20260924 07:09:57.559536 5056 leader_election.cc:288] T 00000000000000000000000000000000 P a448988e73cf47b6bd0e528b8b5d9bac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:09:57.559566 5061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a448988e73cf47b6bd0e528b8b5d9bac [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:09:57.559726 5061 raft_consensus.cc:686] T 00000000000000000000000000000000 P a448988e73cf47b6bd0e528b8b5d9bac [term 1 LEADER]: Becoming Leader. State: Replica: a448988e73cf47b6bd0e528b8b5d9bac, State: Running, Role: LEADER
I20260924 07:09:57.559799 5056 sys_catalog.cc:564] T 00000000000000000000000000000000 P a448988e73cf47b6bd0e528b8b5d9bac [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:09:57.559828 5061 consensus_queue.cc:237] T 00000000000000000000000000000000 P a448988e73cf47b6bd0e528b8b5d9bac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a448988e73cf47b6bd0e528b8b5d9bac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35741 } }
I20260924 07:09:57.560209 5063 sys_catalog.cc:455] T 00000000000000000000000000000000 P a448988e73cf47b6bd0e528b8b5d9bac [sys.catalog]: SysCatalogTable state changed. Reason: New leader a448988e73cf47b6bd0e528b8b5d9bac. Latest consensus state: current_term: 1 leader_uuid: "a448988e73cf47b6bd0e528b8b5d9bac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a448988e73cf47b6bd0e528b8b5d9bac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35741 } } }
I20260924 07:09:57.560215 5062 sys_catalog.cc:455] T 00000000000000000000000000000000 P a448988e73cf47b6bd0e528b8b5d9bac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a448988e73cf47b6bd0e528b8b5d9bac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a448988e73cf47b6bd0e528b8b5d9bac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35741 } } }
I20260924 07:09:57.560357 5063 sys_catalog.cc:458] T 00000000000000000000000000000000 P a448988e73cf47b6bd0e528b8b5d9bac [sys.catalog]: This master's current role is: LEADER
I20260924 07:09:57.560364 5062 sys_catalog.cc:458] T 00000000000000000000000000000000 P a448988e73cf47b6bd0e528b8b5d9bac [sys.catalog]: This master's current role is: LEADER
I20260924 07:09:57.560825 5070 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:09:57.561277 5070 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:09:57.562603 5070 catalog_manager.cc:1347] Generated new cluster ID: 9d9cd87eedfe49c58658fee8c16a5f91
I20260924 07:09:57.562659 5070 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:09:57.577813 5070 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:09:57.577972 5070 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:09:57.581875 5070 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a448988e73cf47b6bd0e528b8b5d9bac: Generated new TSK 0
I20260924 07:09:57.582036 5070 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:09:57.583150 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35741
--builtin_ntp_servers=127.0.98.20:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 07:09:57.672178 5080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:57.672361 5080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:57.672382 5080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:57.673516 5080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:57.673552 5080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:57.673584 5080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:09:57.674871 5080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:57.675096 5080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:57.675386 5080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:57.681227 5088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:57.681237 5086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:57.681236 5085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:57.681387 5080 server_base.cc:1034] running on GCE node
I20260924 07:09:57.681681 5080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:57.681870 5080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:57.683025 5080 hybrid_clock.cc:648] HybridClock initialized: now 1790233797683012 us; error 27 us; skew 500 ppm
I20260924 07:09:57.684278 5080 webserver.cc:460] Webserver started at http://127.0.98.1:35141/ using document root <none> and password file <none>
I20260924 07:09:57.684525 5080 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:57.684569 5080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:57.684715 5080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:57.685756 5080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-0/data/instance:
uuid: "790067f32e0a47e4b59713642f9d2e1e"
format_stamp: "Formatted at 2026-09-24 07:09:57 on dist-test-slave-14ww"
server_key: "922992551a4a2150a318948ea95d48f6"
server_key_iv: "2b6b34a24670156674ed61612bfde8a8"
server_key_version: "encryptionkey@0"
I20260924 07:09:57.686105 5080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-0/wal/instance:
uuid: "790067f32e0a47e4b59713642f9d2e1e"
format_stamp: "Formatted at 2026-09-24 07:09:57 on dist-test-slave-14ww"
server_key: "922992551a4a2150a318948ea95d48f6"
server_key_iv: "2b6b34a24670156674ed61612bfde8a8"
server_key_version: "encryptionkey@0"
I20260924 07:09:57.687381 5080 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:09:57.688133 5094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:57.688282 5080 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:09:57.688381 5080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-0/wal
uuid: "790067f32e0a47e4b59713642f9d2e1e"
format_stamp: "Formatted at 2026-09-24 07:09:57 on dist-test-slave-14ww"
server_key: "922992551a4a2150a318948ea95d48f6"
server_key_iv: "2b6b34a24670156674ed61612bfde8a8"
server_key_version: "encryptionkey@0"
I20260924 07:09:57.688447 5080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:57.691661 5080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:57.691926 5080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:57.692056 5080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:57.692293 5080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:57.692644 5080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:57.692685 5080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:57.692704 5080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:57.692762 5080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:57.699213 5080 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41763
I20260924 07:09:57.699246 5207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41763 every 8 connection(s)
I20260924 07:09:57.699553 5080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:09:57.702334 5059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.700255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45191 (local address 127.0.98.62:35741)
0924 07:09:57.700316 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:57.700317 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:57.700502 (+ 185us) server_negotiation.cc:407] Connection header received
0924 07:09:57.701028 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:57.701029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:57.701033 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:57.701105 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:57.702139 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:57.702140 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:57.702140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:09:57.702144 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:57.702157 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:09:57.702161 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:57.702207 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:57.702272 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 07:09:57.702282 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 07:09:57.702497 5210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.700247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35741 (local address 127.0.98.1:45191)
0924 07:09:57.700408 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:57.700438 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:57.700944 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:57.701157 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:57.701159 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:57.701167 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:57.702001 (+ 834us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:09:57.702002 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:57.702036 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:09:57.702062 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:57.702213 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:57.702215 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:57.702218 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:09:57.702269 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 07:09:57.702312 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260924 07:09:57.703058 5208 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35741
I20260924 07:09:57.703141 5208 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:57.703299 5208 heartbeater.cc:511] Master 127.0.98.62:35741 requested a full tablet report, sending...
I20260924 07:09:57.703761 5020 ts_manager.cc:194] Registered new tserver with Master: 790067f32e0a47e4b59713642f9d2e1e (127.0.98.1:41763)
I20260924 07:09:57.708089 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 5080
I20260924 07:09:57.708204 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:09:57.708375 392 external_mini_cluster.cc:1428] Setting key b803b87f30600b7a8932bea4837762dc
I20260924 07:09:57.709252 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35741
--builtin_ntp_servers=127.0.98.20:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 07:09:57.788072 5211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:57.788295 5211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:57.788331 5211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:57.789551 5211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:57.789616 5211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:57.789661 5211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:09:57.790993 5211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:57.791263 5211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:57.791551 5211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:57.796288 5216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:57.796293 5219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:57.796353 5217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:57.796470 5211 server_base.cc:1034] running on GCE node
I20260924 07:09:57.797433 5211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:57.797690 5211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:57.798846 5211 hybrid_clock.cc:648] HybridClock initialized: now 1790233797798826 us; error 37 us; skew 500 ppm
I20260924 07:09:57.799940 5211 webserver.cc:460] Webserver started at http://127.0.98.2:33373/ using document root <none> and password file <none>
I20260924 07:09:57.800128 5211 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:57.800169 5211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:57.800256 5211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:57.801293 5211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-1/data/instance:
uuid: "c1cecc63913e42caa2c758f39f7b44d2"
format_stamp: "Formatted at 2026-09-24 07:09:57 on dist-test-slave-14ww"
server_key: "c45bc56f5e8d7f0b40197342f1bcb571"
server_key_iv: "5acb0be00e14639f6a808358eba4d188"
server_key_version: "encryptionkey@0"
I20260924 07:09:57.801595 5211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c1cecc63913e42caa2c758f39f7b44d2"
format_stamp: "Formatted at 2026-09-24 07:09:57 on dist-test-slave-14ww"
server_key: "c45bc56f5e8d7f0b40197342f1bcb571"
server_key_iv: "5acb0be00e14639f6a808358eba4d188"
server_key_version: "encryptionkey@0"
I20260924 07:09:57.802757 5211 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 07:09:57.803392 5225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:57.803551 5211 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:09:57.803638 5211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-1/wal
uuid: "c1cecc63913e42caa2c758f39f7b44d2"
format_stamp: "Formatted at 2026-09-24 07:09:57 on dist-test-slave-14ww"
server_key: "c45bc56f5e8d7f0b40197342f1bcb571"
server_key_iv: "5acb0be00e14639f6a808358eba4d188"
server_key_version: "encryptionkey@0"
I20260924 07:09:57.803694 5211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:57.806829 5211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:57.807017 5211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:57.807114 5211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:57.807251 5211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:57.807468 5211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:57.807497 5211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:57.807516 5211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:57.807528 5211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:57.812698 5211 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36607
I20260924 07:09:57.812789 5338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36607 every 8 connection(s)
I20260924 07:09:57.813139 5211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:09:57.815928 5059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.813807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57897 (local address 127.0.98.62:35741)
0924 07:09:57.813889 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:57.813891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:57.814081 (+ 190us) server_negotiation.cc:407] Connection header received
0924 07:09:57.814547 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:57.814549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:57.814555 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:57.814605 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:57.815698 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:57.815699 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:57.815700 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:09:57.815704 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:57.815719 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:09:57.815724 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:57.815776 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:57.815845 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 07:09:57.815860 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 07:09:57.816107 5341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.813780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35741 (local address 127.0.98.2:57897)
0924 07:09:57.813965 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:57.814000 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:57.814437 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:57.814678 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:57.814682 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:57.814691 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:57.815552 (+ 861us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:09:57.815552 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:57.815578 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:09:57.815602 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:57.815773 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:57.815777 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:57.815780 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:09:57.815845 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 07:09:57.815901 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":78,"threads_started":1}
I20260924 07:09:57.816555 5339 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35741
I20260924 07:09:57.816627 5339 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:57.816828 5339 heartbeater.cc:511] Master 127.0.98.62:35741 requested a full tablet report, sending...
I20260924 07:09:57.817178 5020 ts_manager.cc:194] Registered new tserver with Master: c1cecc63913e42caa2c758f39f7b44d2 (127.0.98.2:36607)
I20260924 07:09:57.823376 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 5211
I20260924 07:09:57.823493 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:09:57.823678 392 external_mini_cluster.cc:1428] Setting key ee71ef4574a755216a335968db969f5b
I20260924 07:09:57.824581 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35741
--builtin_ntp_servers=127.0.98.20:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 07:09:57.903077 5342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:57.903306 5342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:57.903340 5342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:57.904497 5342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:57.904561 5342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:57.904605 5342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:09:57.906067 5342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:57.906317 5342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:57.906596 5342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:57.911540 5347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:57.911490 5350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:57.911540 5348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:57.911736 5342 server_base.cc:1034] running on GCE node
I20260924 07:09:57.911985 5342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:57.912225 5342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:57.913383 5342 hybrid_clock.cc:648] HybridClock initialized: now 1790233797913384 us; error 39 us; skew 500 ppm
I20260924 07:09:57.914672 5342 webserver.cc:460] Webserver started at http://127.0.98.3:40007/ using document root <none> and password file <none>
I20260924 07:09:57.914913 5342 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:57.915002 5342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:57.915138 5342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:57.916133 5342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-2/data/instance:
uuid: "3da6d5c549504ceca9e6fbadd789d073"
format_stamp: "Formatted at 2026-09-24 07:09:57 on dist-test-slave-14ww"
server_key: "8c3151c14b8f4582435c9ff0816d3d80"
server_key_iv: "64e2be56310b7c71cc22ac8f11f90264"
server_key_version: "encryptionkey@0"
I20260924 07:09:57.916492 5342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3da6d5c549504ceca9e6fbadd789d073"
format_stamp: "Formatted at 2026-09-24 07:09:57 on dist-test-slave-14ww"
server_key: "8c3151c14b8f4582435c9ff0816d3d80"
server_key_iv: "64e2be56310b7c71cc22ac8f11f90264"
server_key_version: "encryptionkey@0"
I20260924 07:09:57.917842 5342 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:09:57.918579 5356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:57.918756 5342 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:09:57.918857 5342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-2/wal
uuid: "3da6d5c549504ceca9e6fbadd789d073"
format_stamp: "Formatted at 2026-09-24 07:09:57 on dist-test-slave-14ww"
server_key: "8c3151c14b8f4582435c9ff0816d3d80"
server_key_iv: "64e2be56310b7c71cc22ac8f11f90264"
server_key_version: "encryptionkey@0"
I20260924 07:09:57.918943 5342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:57.922219 5342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:57.922467 5342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:57.922621 5342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:57.922821 5342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:09:57.923106 5342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:09:57.923170 5342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:57.923209 5342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:09:57.923237 5342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:57.928830 5342 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44049
I20260924 07:09:57.928889 5469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44049 every 8 connection(s)
I20260924 07:09:57.929183 5342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:09:57.932245 5059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.929850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55845 (local address 127.0.98.62:35741)
0924 07:09:57.929943 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:57.929946 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:57.930153 (+ 207us) server_negotiation.cc:407] Connection header received
0924 07:09:57.930694 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:57.930696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:57.930702 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:57.930760 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:57.931963 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:57.931964 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:57.931964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:09:57.931969 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:57.931985 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:09:57.931990 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:57.932042 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:57.932160 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 07:09:57.932176 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 07:09:57.932439 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.929826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35741 (local address 127.0.98.3:55845)
0924 07:09:57.930023 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:57.930055 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:57.930589 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:57.930829 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:57.930839 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:57.930847 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:57.931805 (+ 958us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:09:57.931805 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:57.931833 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:09:57.931860 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:57.932056 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:57.932061 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:57.932066 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:09:57.932167 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 07:09:57.932228 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
I20260924 07:09:57.932986 5470 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35741
I20260924 07:09:57.933077 5470 heartbeater.cc:464] Registering TS with master...
I20260924 07:09:57.933259 5470 heartbeater.cc:511] Master 127.0.98.62:35741 requested a full tablet report, sending...
I20260924 07:09:57.933619 5020 ts_manager.cc:194] Registered new tserver with Master: 3da6d5c549504ceca9e6fbadd789d073 (127.0.98.3:44049)
I20260924 07:09:57.938787 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 5342
I20260924 07:09:57.938911 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:09:57.939105 392 external_mini_cluster.cc:1428] Setting key a61b7beb61a56fa86976b5daab4717aa
I20260924 07:09:57.939760 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:09:57.942186 5059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.941115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38694 (local address 127.0.98.62:35741)
0924 07:09:57.941171 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:57.941173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:57.941278 (+ 105us) server_negotiation.cc:407] Connection header received
0924 07:09:57.941313 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:57.941313 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:57.941317 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:57.941376 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:57.941962 (+ 586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:57.941963 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:57.941963 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:09:57.941967 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:57.941978 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:09:57.941981 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:57.942032 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:57.942067 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 07:09:57.942078 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 07:09:57.943634 5020 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38694:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:09:57.945283 5020 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:09:57.954634 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.952462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44049 (local address 127.0.0.1:49032)
0924 07:09:57.952577 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:57.952632 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:57.952690 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:57.953558 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:57.953564 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:57.953571 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:57.954152 (+ 581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:09:57.954153 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:57.954169 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:09:57.954195 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:57.954482 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:57.954488 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:57.954493 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:09:57.954540 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 07:09:57.954553 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":64,"threads_started":1}
I20260924 07:09:57.954993 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.952470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41763 (local address 127.0.0.1:50022)
0924 07:09:57.953860 (+ 1390us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:57.953869 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:57.953914 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:57.954051 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:57.954052 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:57.954054 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:57.954526 (+ 472us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:09:57.954527 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:57.954534 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:09:57.954546 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:57.954867 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:57.954868 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:57.954868 (+ 0us) client_negotiation.cc:769] Sending connection context
0924 07:09:57.954917 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 07:09:57.954929 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":997,"thread_start_us":83,"threads_started":1}
I20260924 07:09:57.955044 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.953210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49032 (local address 127.0.98.3:44049)
0924 07:09:57.953397 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:57.953402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:57.953413 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:09:57.953460 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:57.953462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:57.953471 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:57.953553 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:57.954323 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:57.954329 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:57.954330 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:09:57.954334 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:57.954366 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:09:57.954388 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:57.954916 (+ 528us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:57.954938 (+ 22us) server_negotiation.cc:299] Negotiation successful
0924 07:09:57.954955 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":115,"threads_started":1}
I20260924 07:09:57.955039 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.952696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50022 (local address 127.0.98.1:41763)
0924 07:09:57.952871 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:57.952876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:57.953924 (+ 1048us) server_negotiation.cc:407] Connection header received
0924 07:09:57.953960 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:57.953963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:57.953972 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:57.954046 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:57.954667 (+ 621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:57.954674 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:57.954675 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:09:57.954681 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:57.954730 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:09:57.954754 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:57.954813 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:57.954938 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 07:09:57.954961 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":103,"threads_started":1}
I20260924 07:09:57.956195 5142 tablet_service.cc:1467] Processing CreateTablet for tablet 5997945f521641f5ac5625faf313cbd8 (DEFAULT_TABLE table=test-table [id=404114aaa5c34553ac2ebcbb22b68647]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:57.956188 5404 tablet_service.cc:1467] Processing CreateTablet for tablet 5997945f521641f5ac5625faf313cbd8 (DEFAULT_TABLE table=test-table [id=404114aaa5c34553ac2ebcbb22b68647]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:57.956554 5142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5997945f521641f5ac5625faf313cbd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:57.956555 5404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5997945f521641f5ac5625faf313cbd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:57.960016 5484 tablet_bootstrap.cc:492] T 5997945f521641f5ac5625faf313cbd8 P 3da6d5c549504ceca9e6fbadd789d073: Bootstrap starting.
I20260924 07:09:57.960455 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.952653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36607 (local address 127.0.0.1:38710)
0924 07:09:57.954710 (+ 2057us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:57.954720 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:57.954758 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:57.955304 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:57.955305 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:57.955307 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:57.960069 (+ 4762us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:09:57.960069 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:57.960075 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:09:57.960116 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:57.960354 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:57.960354 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:57.960356 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 07:09:57.960383 (+ 27us) client_negotiation.cc:240] Negotiation successful
0924 07:09:57.960397 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2048,"thread_start_us":67,"threads_started":1}
I20260924 07:09:57.960431 5485 tablet_bootstrap.cc:492] T 5997945f521641f5ac5625faf313cbd8 P 790067f32e0a47e4b59713642f9d2e1e: Bootstrap starting.
I20260924 07:09:57.960768 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:57.952917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38710 (local address 127.0.98.2:36607)
0924 07:09:57.953060 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:57.953064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:57.955134 (+ 2070us) server_negotiation.cc:407] Connection header received
0924 07:09:57.955166 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:57.955169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:57.955180 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:57.955254 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:57.960203 (+ 4949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:57.960211 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:57.960211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:09:57.960215 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:57.960249 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:09:57.960278 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:57.960626 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:57.960652 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 07:09:57.960667 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":76,"threads_started":1}
I20260924 07:09:57.960788 5484 tablet_bootstrap.cc:654] T 5997945f521641f5ac5625faf313cbd8 P 3da6d5c549504ceca9e6fbadd789d073: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:57.961118 5484 log.cc:824] T 5997945f521641f5ac5625faf313cbd8 P 3da6d5c549504ceca9e6fbadd789d073: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:57.961565 5485 tablet_bootstrap.cc:654] T 5997945f521641f5ac5625faf313cbd8 P 790067f32e0a47e4b59713642f9d2e1e: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:57.961463 5273 tablet_service.cc:1467] Processing CreateTablet for tablet 5997945f521641f5ac5625faf313cbd8 (DEFAULT_TABLE table=test-table [id=404114aaa5c34553ac2ebcbb22b68647]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:09:57.961776 5484 tablet_bootstrap.cc:492] T 5997945f521641f5ac5625faf313cbd8 P 3da6d5c549504ceca9e6fbadd789d073: No bootstrap required, opened a new log
I20260924 07:09:57.961777 5273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5997945f521641f5ac5625faf313cbd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:57.961829 5484 ts_tablet_manager.cc:1397] T 5997945f521641f5ac5625faf313cbd8 P 3da6d5c549504ceca9e6fbadd789d073: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 07:09:57.962045 5485 log.cc:824] T 5997945f521641f5ac5625faf313cbd8 P 790067f32e0a47e4b59713642f9d2e1e: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:57.963181 5485 tablet_bootstrap.cc:492] T 5997945f521641f5ac5625faf313cbd8 P 790067f32e0a47e4b59713642f9d2e1e: No bootstrap required, opened a new log
I20260924 07:09:57.963295 5485 ts_tablet_manager.cc:1397] T 5997945f521641f5ac5625faf313cbd8 P 790067f32e0a47e4b59713642f9d2e1e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 07:09:57.963811 5484 raft_consensus.cc:348] T 5997945f521641f5ac5625faf313cbd8 P 3da6d5c549504ceca9e6fbadd789d073 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da6d5c549504ceca9e6fbadd789d073" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44049 } } peers { permanent_uuid: "c1cecc63913e42caa2c758f39f7b44d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36607 } } peers { permanent_uuid: "790067f32e0a47e4b59713642f9d2e1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41763 } }
I20260924 07:09:57.963935 5484 raft_consensus.cc:374] T 5997945f521641f5ac5625faf313cbd8 P 3da6d5c549504ceca9e6fbadd789d073 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:57.963956 5484 raft_consensus.cc:729] T 5997945f521641f5ac5625faf313cbd8 P 3da6d5c549504ceca9e6fbadd789d073 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3da6d5c549504ceca9e6fbadd789d073, State: Initialized, Role: FOLLOWER
I20260924 07:09:57.964071 5484 consensus_queue.cc:260] T 5997945f521641f5ac5625faf313cbd8 P 3da6d5c549504ceca9e6fbadd789d073 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da6d5c549504ceca9e6fbadd789d073" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44049 } } peers { permanent_uuid: "c1cecc63913e42caa2c758f39f7b44d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36607 } } peers { permanent_uuid: "790067f32e0a47e4b59713642f9d2e1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41763 } }
I20260924 07:09:57.964324 5484 ts_tablet_manager.cc:1428] T 5997945f521641f5ac5625faf313cbd8 P 3da6d5c549504ceca9e6fbadd789d073: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 07:09:57.964339 5470 heartbeater.cc:503] Master 127.0.98.62:35741 was elected leader, sending a full tablet report...
I20260924 07:09:57.964931 5489 tablet_bootstrap.cc:492] T 5997945f521641f5ac5625faf313cbd8 P c1cecc63913e42caa2c758f39f7b44d2: Bootstrap starting.
I20260924 07:09:57.965674 5489 tablet_bootstrap.cc:654] T 5997945f521641f5ac5625faf313cbd8 P c1cecc63913e42caa2c758f39f7b44d2: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:57.965950 5489 log.cc:824] T 5997945f521641f5ac5625faf313cbd8 P c1cecc63913e42caa2c758f39f7b44d2: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:57.966573 5489 tablet_bootstrap.cc:492] T 5997945f521641f5ac5625faf313cbd8 P c1cecc63913e42caa2c758f39f7b44d2: No bootstrap required, opened a new log
I20260924 07:09:57.966622 5489 ts_tablet_manager.cc:1397] T 5997945f521641f5ac5625faf313cbd8 P c1cecc63913e42caa2c758f39f7b44d2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:09:57.966665 5485 raft_consensus.cc:348] T 5997945f521641f5ac5625faf313cbd8 P 790067f32e0a47e4b59713642f9d2e1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da6d5c549504ceca9e6fbadd789d073" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44049 } } peers { permanent_uuid: "c1cecc63913e42caa2c758f39f7b44d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36607 } } peers { permanent_uuid: "790067f32e0a47e4b59713642f9d2e1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41763 } }
I20260924 07:09:57.966816 5485 raft_consensus.cc:374] T 5997945f521641f5ac5625faf313cbd8 P 790067f32e0a47e4b59713642f9d2e1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:57.966862 5485 raft_consensus.cc:729] T 5997945f521641f5ac5625faf313cbd8 P 790067f32e0a47e4b59713642f9d2e1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 790067f32e0a47e4b59713642f9d2e1e, State: Initialized, Role: FOLLOWER
I20260924 07:09:57.966976 5485 consensus_queue.cc:260] T 5997945f521641f5ac5625faf313cbd8 P 790067f32e0a47e4b59713642f9d2e1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da6d5c549504ceca9e6fbadd789d073" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44049 } } peers { permanent_uuid: "c1cecc63913e42caa2c758f39f7b44d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36607 } } peers { permanent_uuid: "790067f32e0a47e4b59713642f9d2e1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41763 } }
I20260924 07:09:57.967213 5485 ts_tablet_manager.cc:1428] T 5997945f521641f5ac5625faf313cbd8 P 790067f32e0a47e4b59713642f9d2e1e: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260924 07:09:57.967249 5208 heartbeater.cc:503] Master 127.0.98.62:35741 was elected leader, sending a full tablet report...
I20260924 07:09:57.968974 5489 raft_consensus.cc:348] T 5997945f521641f5ac5625faf313cbd8 P c1cecc63913e42caa2c758f39f7b44d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da6d5c549504ceca9e6fbadd789d073" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44049 } } peers { permanent_uuid: "c1cecc63913e42caa2c758f39f7b44d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36607 } } peers { permanent_uuid: "790067f32e0a47e4b59713642f9d2e1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41763 } }
I20260924 07:09:57.969089 5489 raft_consensus.cc:374] T 5997945f521641f5ac5625faf313cbd8 P c1cecc63913e42caa2c758f39f7b44d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:57.969110 5489 raft_consensus.cc:729] T 5997945f521641f5ac5625faf313cbd8 P c1cecc63913e42caa2c758f39f7b44d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1cecc63913e42caa2c758f39f7b44d2, State: Initialized, Role: FOLLOWER
I20260924 07:09:57.969218 5489 consensus_queue.cc:260] T 5997945f521641f5ac5625faf313cbd8 P c1cecc63913e42caa2c758f39f7b44d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da6d5c549504ceca9e6fbadd789d073" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44049 } } peers { permanent_uuid: "c1cecc63913e42caa2c758f39f7b44d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36607 } } peers { permanent_uuid: "790067f32e0a47e4b59713642f9d2e1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41763 } }
I20260924 07:09:57.969419 5489 ts_tablet_manager.cc:1428] T 5997945f521641f5ac5625faf313cbd8 P c1cecc63913e42caa2c758f39f7b44d2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 07:09:57.969481 5339 heartbeater.cc:503] Master 127.0.98.62:35741 was elected leader, sending a full tablet report...
I20260924 07:09:58.010573 5491 raft_consensus.cc:482] T 5997945f521641f5ac5625faf313cbd8 P 790067f32e0a47e4b59713642f9d2e1e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:09:58.010712 5491 raft_consensus.cc:504] T 5997945f521641f5ac5625faf313cbd8 P 790067f32e0a47e4b59713642f9d2e1e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da6d5c549504ceca9e6fbadd789d073" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44049 } } peers { permanent_uuid: "c1cecc63913e42caa2c758f39f7b44d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36607 } } peers { permanent_uuid: "790067f32e0a47e4b59713642f9d2e1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41763 } }
I20260924 07:09:58.011004 5491 leader_election.cc:288] T 5997945f521641f5ac5625faf313cbd8 P 790067f32e0a47e4b59713642f9d2e1e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3da6d5c549504ceca9e6fbadd789d073 (127.0.98.3:44049), c1cecc63913e42caa2c758f39f7b44d2 (127.0.98.2:36607)
I20260924 07:09:58.012550 5210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:58.011246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44049 (local address 127.0.98.1:39903)
0924 07:09:58.011334 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:58.011347 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:58.011404 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:58.011653 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:58.011654 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:58.011657 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:58.012192 (+ 535us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:09:58.012192 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:58.012199 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:09:58.012210 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:58.012392 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:58.012392 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:58.012393 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 07:09:58.012445 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 07:09:58.012459 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 07:09:58.012678 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:58.011339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39903 (local address 127.0.98.3:44049)
0924 07:09:58.011496 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:58.011499 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:58.011507 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:09:58.011536 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:58.011537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:58.011542 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:58.011595 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:58.012296 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:58.012297 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:58.012298 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:09:58.012302 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:58.012324 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:09:58.012328 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:58.012394 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:58.012563 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 07:09:58.012577 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260924 07:09:58.012702 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:58.011377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36607 (local address 127.0.98.1:55673)
0924 07:09:58.011559 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:58.011570 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:58.011669 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:58.011864 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:58.011865 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:58.011868 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:09:58.012362 (+ 494us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:09:58.012363 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:09:58.012370 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:09:58.012382 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:09:58.012537 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:09:58.012538 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:09:58.012538 (+ 0us) client_negotiation.cc:769] Sending connection context
0924 07:09:58.012606 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 07:09:58.012620 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":105,"threads_started":1}
I20260924 07:09:58.012691 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:58.011549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55673 (local address 127.0.98.2:36607)
0924 07:09:58.011594 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:58.011596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:58.011628 (+ 32us) server_negotiation.cc:407] Connection header received
0924 07:09:58.011767 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:58.011768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:58.011774 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:58.011837 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:58.012460 (+ 623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:58.012461 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:58.012461 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:09:58.012465 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:58.012478 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:09:58.012482 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:58.012537 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:58.012610 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 07:09:58.012625 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 07:09:58.012995 5424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5997945f521641f5ac5625faf313cbd8" candidate_uuid: "790067f32e0a47e4b59713642f9d2e1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3da6d5c549504ceca9e6fbadd789d073" is_pre_election: true
I20260924 07:09:58.013142 5424 raft_consensus.cc:2454] T 5997945f521641f5ac5625faf313cbd8 P 3da6d5c549504ceca9e6fbadd789d073 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 790067f32e0a47e4b59713642f9d2e1e in term 0.
I20260924 07:09:58.013196 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5997945f521641f5ac5625faf313cbd8" candidate_uuid: "790067f32e0a47e4b59713642f9d2e1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1cecc63913e42caa2c758f39f7b44d2" is_pre_election: true
I20260924 07:09:58.013347 5095 leader_election.cc:302] T 5997945f521641f5ac5625faf313cbd8 P 790067f32e0a47e4b59713642f9d2e1e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3da6d5c549504ceca9e6fbadd789d073, 790067f32e0a47e4b59713642f9d2e1e; no voters:
I20260924 07:09:58.013356 5293 raft_consensus.cc:2454] T 5997945f521641f5ac5625faf313cbd8 P c1cecc63913e42caa2c758f39f7b44d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 790067f32e0a47e4b59713642f9d2e1e in term 0.
I20260924 07:09:58.013463 5491 raft_consensus.cc:2781] T 5997945f521641f5ac5625faf313cbd8 P 790067f32e0a47e4b59713642f9d2e1e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:09:58.013533 5491 raft_consensus.cc:482] T 5997945f521641f5ac5625faf313cbd8 P 790067f32e0a47e4b59713642f9d2e1e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:09:58.013569 5491 raft_consensus.cc:3037] T 5997945f521641f5ac5625faf313cbd8 P 790067f32e0a47e4b59713642f9d2e1e [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:58.014302 5491 raft_consensus.cc:504] T 5997945f521641f5ac5625faf313cbd8 P 790067f32e0a47e4b59713642f9d2e1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da6d5c549504ceca9e6fbadd789d073" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44049 } } peers { permanent_uuid: "c1cecc63913e42caa2c758f39f7b44d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36607 } } peers { permanent_uuid: "790067f32e0a47e4b59713642f9d2e1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41763 } }
I20260924 07:09:58.014441 5491 leader_election.cc:288] T 5997945f521641f5ac5625faf313cbd8 P 790067f32e0a47e4b59713642f9d2e1e [CANDIDATE]: Term 1 election: Requested vote from peers 3da6d5c549504ceca9e6fbadd789d073 (127.0.98.3:44049), c1cecc63913e42caa2c758f39f7b44d2 (127.0.98.2:36607)
I20260924 07:09:58.014603 5424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5997945f521641f5ac5625faf313cbd8" candidate_uuid: "790067f32e0a47e4b59713642f9d2e1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3da6d5c549504ceca9e6fbadd789d073"
I20260924 07:09:58.014683 5424 raft_consensus.cc:3037] T 5997945f521641f5ac5625faf313cbd8 P 3da6d5c549504ceca9e6fbadd789d073 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:58.014789 5293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5997945f521641f5ac5625faf313cbd8" candidate_uuid: "790067f32e0a47e4b59713642f9d2e1e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1cecc63913e42caa2c758f39f7b44d2"
I20260924 07:09:58.014861 5293 raft_consensus.cc:3037] T 5997945f521641f5ac5625faf313cbd8 P c1cecc63913e42caa2c758f39f7b44d2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:58.015413 5424 raft_consensus.cc:2454] T 5997945f521641f5ac5625faf313cbd8 P 3da6d5c549504ceca9e6fbadd789d073 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 790067f32e0a47e4b59713642f9d2e1e in term 1.
I20260924 07:09:58.015607 5293 raft_consensus.cc:2454] T 5997945f521641f5ac5625faf313cbd8 P c1cecc63913e42caa2c758f39f7b44d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 790067f32e0a47e4b59713642f9d2e1e in term 1.
I20260924 07:09:58.015622 5095 leader_election.cc:302] T 5997945f521641f5ac5625faf313cbd8 P 790067f32e0a47e4b59713642f9d2e1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3da6d5c549504ceca9e6fbadd789d073, 790067f32e0a47e4b59713642f9d2e1e; no voters:
I20260924 07:09:58.015734 5491 raft_consensus.cc:2781] T 5997945f521641f5ac5625faf313cbd8 P 790067f32e0a47e4b59713642f9d2e1e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:09:58.015890 5491 raft_consensus.cc:686] T 5997945f521641f5ac5625faf313cbd8 P 790067f32e0a47e4b59713642f9d2e1e [term 1 LEADER]: Becoming Leader. State: Replica: 790067f32e0a47e4b59713642f9d2e1e, State: Running, Role: LEADER
I20260924 07:09:58.015985 5491 consensus_queue.cc:237] T 5997945f521641f5ac5625faf313cbd8 P 790067f32e0a47e4b59713642f9d2e1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da6d5c549504ceca9e6fbadd789d073" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44049 } } peers { permanent_uuid: "c1cecc63913e42caa2c758f39f7b44d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36607 } } peers { permanent_uuid: "790067f32e0a47e4b59713642f9d2e1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41763 } }
I20260924 07:09:58.016952 5020 catalog_manager.cc:5495] T 5997945f521641f5ac5625faf313cbd8 P 790067f32e0a47e4b59713642f9d2e1e reported cstate change: term changed from 0 to 1, leader changed from <none> to 790067f32e0a47e4b59713642f9d2e1e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "790067f32e0a47e4b59713642f9d2e1e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3da6d5c549504ceca9e6fbadd789d073" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1cecc63913e42caa2c758f39f7b44d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "790067f32e0a47e4b59713642f9d2e1e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41763 } health_report { overall_health: HEALTHY } } }
I20260924 07:09:58.029390 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:58.028416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50030 (local address 127.0.98.1:41763)
0924 07:09:58.028472 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:58.028474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:58.028483 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:09:58.028559 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:58.028564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:58.028575 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:58.028638 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:58.029177 (+ 539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:58.029178 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:58.029179 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:09:58.029183 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:58.029196 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:09:58.029200 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:58.029256 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:58.029311 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 07:09:58.029324 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 07:09:58.030926 5293 raft_consensus.cc:1261] T 5997945f521641f5ac5625faf313cbd8 P c1cecc63913e42caa2c758f39f7b44d2 [term 1 FOLLOWER]: Refusing update from remote peer 790067f32e0a47e4b59713642f9d2e1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:09:58.031009 5424 raft_consensus.cc:1261] T 5997945f521641f5ac5625faf313cbd8 P 3da6d5c549504ceca9e6fbadd789d073 [term 1 FOLLOWER]: Refusing update from remote peer 790067f32e0a47e4b59713642f9d2e1e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:09:58.031232 5491 consensus_queue.cc:1035] T 5997945f521641f5ac5625faf313cbd8 P 790067f32e0a47e4b59713642f9d2e1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1cecc63913e42caa2c758f39f7b44d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:09:58.031378 5494 consensus_queue.cc:1035] T 5997945f521641f5ac5625faf313cbd8 P 790067f32e0a47e4b59713642f9d2e1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3da6d5c549504ceca9e6fbadd789d073" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:09:58.033416 5497 mvcc.cc:205] Tried to move back new op lower bound from 7332797636732334080 to 7332797636674781184. Current Snapshot: MvccSnapshot[applied={T|T < 7332797636732334080}]
I20260924 07:09:58.033643 5499 mvcc.cc:205] Tried to move back new op lower bound from 7332797636732334080 to 7332797636674781184. Current Snapshot: MvccSnapshot[applied={T|T < 7332797636732334080}]
I20260924 07:09:58.034027 5498 mvcc.cc:205] Tried to move back new op lower bound from 7332797636732334080 to 7332797636674781184. Current Snapshot: MvccSnapshot[applied={T|T < 7332797636732334080}]
I20260924 07:09:58.035562 5019 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:09:58.035634 5019 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:09:58.036324 5019 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a448988e73cf47b6bd0e528b8b5d9bac: Sending DeleteTablet for 3 replicas of tablet 5997945f521641f5ac5625faf313cbd8
I20260924 07:09:58.036629 5142 tablet_service.cc:1514] Processing DeleteTablet for tablet 5997945f521641f5ac5625faf313cbd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:58 UTC) from {username='slave'} at 127.0.0.1:50022
I20260924 07:09:58.036684 5404 tablet_service.cc:1514] Processing DeleteTablet for tablet 5997945f521641f5ac5625faf313cbd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:58 UTC) from {username='slave'} at 127.0.0.1:49032
I20260924 07:09:58.036725 5273 tablet_service.cc:1514] Processing DeleteTablet for tablet 5997945f521641f5ac5625faf313cbd8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:09:58 UTC) from {username='slave'} at 127.0.0.1:38710
I20260924 07:09:58.036880 5508 tablet_replica.cc:330] T 5997945f521641f5ac5625faf313cbd8 P 3da6d5c549504ceca9e6fbadd789d073: stopping tablet replica
I20260924 07:09:58.036964 5508 raft_consensus.cc:2229] T 5997945f521641f5ac5625faf313cbd8 P 3da6d5c549504ceca9e6fbadd789d073 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:09:58.037066 5508 raft_consensus.cc:2258] T 5997945f521641f5ac5625faf313cbd8 P 3da6d5c549504ceca9e6fbadd789d073 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:09:58.037087 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 5080
I20260924 07:09:58.037642 5509 tablet_replica.cc:330] T 5997945f521641f5ac5625faf313cbd8 P c1cecc63913e42caa2c758f39f7b44d2: stopping tablet replica
I20260924 07:09:58.037742 5509 raft_consensus.cc:2229] T 5997945f521641f5ac5625faf313cbd8 P c1cecc63913e42caa2c758f39f7b44d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:09:58.037843 5509 raft_consensus.cc:2258] T 5997945f521641f5ac5625faf313cbd8 P c1cecc63913e42caa2c758f39f7b44d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:09:58.038105 5508 ts_tablet_manager.cc:1905] T 5997945f521641f5ac5625faf313cbd8 P 3da6d5c549504ceca9e6fbadd789d073: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:09:58.039321 5509 ts_tablet_manager.cc:1905] T 5997945f521641f5ac5625faf313cbd8 P c1cecc63913e42caa2c758f39f7b44d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:09:58.039825 5508 ts_tablet_manager.cc:1918] T 5997945f521641f5ac5625faf313cbd8 P 3da6d5c549504ceca9e6fbadd789d073: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:09:58.039896 5508 log.cc:1196] T 5997945f521641f5ac5625faf313cbd8 P 3da6d5c549504ceca9e6fbadd789d073: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-2/wal/wals/5997945f521641f5ac5625faf313cbd8
I20260924 07:09:58.040153 5508 ts_tablet_manager.cc:1939] T 5997945f521641f5ac5625faf313cbd8 P 3da6d5c549504ceca9e6fbadd789d073: Deleting consensus metadata
I20260924 07:09:58.040555 5004 catalog_manager.cc:4841] TS 3da6d5c549504ceca9e6fbadd789d073 (127.0.98.3:44049): tablet 5997945f521641f5ac5625faf313cbd8 (table test-table [id=404114aaa5c34553ac2ebcbb22b68647]) successfully deleted
I20260924 07:09:58.040992 5509 ts_tablet_manager.cc:1918] T 5997945f521641f5ac5625faf313cbd8 P c1cecc63913e42caa2c758f39f7b44d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:09:58.041047 5509 log.cc:1196] T 5997945f521641f5ac5625faf313cbd8 P c1cecc63913e42caa2c758f39f7b44d2: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790233772407728-392-0/minicluster-data/ts-1/wal/wals/5997945f521641f5ac5625faf313cbd8
I20260924 07:09:58.041289 5509 ts_tablet_manager.cc:1939] T 5997945f521641f5ac5625faf313cbd8 P c1cecc63913e42caa2c758f39f7b44d2: Deleting consensus metadata
I20260924 07:09:58.041785 5004 catalog_manager.cc:4841] TS c1cecc63913e42caa2c758f39f7b44d2 (127.0.98.2:36607): tablet 5997945f521641f5ac5625faf313cbd8 (table test-table [id=404114aaa5c34553ac2ebcbb22b68647]) successfully deleted
W20260924 07:09:58.043015 5007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41763 (error 108)
I20260924 07:09:58.043489 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:58.043368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41763 (local address 127.0.0.1:50034)
0924 07:09:58.043418 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:58.043455 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 07:09:58.043514 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 5211
W20260924 07:09:58.043598 5007 catalog_manager.cc:4584] TS 790067f32e0a47e4b59713642f9d2e1e (127.0.98.1:41763): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5997945f521641f5ac5625faf313cbd8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41763: connect: Connection refused (error 111)
I20260924 07:09:58.048589 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 5342
I20260924 07:09:58.052528 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 4989
2026-09-24T07:09:58Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (628 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 07:09:58.071484 392 mini_oidc.cc:152] Starting JWKS server
I20260924 07:09:58.072512 392 webserver.cc:460] Webserver started at https://127.0.0.1:33629/ using document root <none> and password file <none>
I20260924 07:09:58.072636 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 07:09:58.072858 392 webserver.cc:460] Webserver started at http://127.0.0.1:38173/ using document root <none> and password file <none>
2026-09-24T07:09:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:09:58Z Disabled control of system clock
I20260924 07:09:58.080106 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37597
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37597
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33629/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233772407728-392-0/testchainca.cert with env {}
W20260924 07:09:58.157984 5516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:58.158223 5516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:58.158258 5516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:58.159397 5516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:09:58.159482 5516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:58.159507 5516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:58.159538 5516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:09:58.159560 5516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 07:09:58.159600 5516 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 07:09:58.159621 5516 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33629/jwks.json
I20260924 07:09:58.161571 5516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37597
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33629/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233772407728-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:58.161831 5516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:58.162060 5516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:58.166873 5524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:58.166888 5522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:58.167002 5516 server_base.cc:1034] running on GCE node
W20260924 07:09:58.166893 5521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:58.167366 5516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:58.167603 5516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:58.168792 5516 hybrid_clock.cc:648] HybridClock initialized: now 1790233798168763 us; error 43 us; skew 500 ppm
I20260924 07:09:58.169960 5516 webserver.cc:460] Webserver started at http://127.0.98.62:44285/ using document root <none> and password file <none>
I20260924 07:09:58.170208 5516 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:58.170255 5516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:58.170406 5516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:58.171413 5516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "0d8048bd1e0747d0be77403cda8c3986"
format_stamp: "Formatted at 2026-09-24 07:09:58 on dist-test-slave-14ww"
server_key: "7fde23b33d77d1e7db38e8a197e68311"
server_key_iv: "5042990a80c3141d5746a50bff682027"
server_key_version: "encryptionkey@0"
I20260924 07:09:58.171788 5516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "0d8048bd1e0747d0be77403cda8c3986"
format_stamp: "Formatted at 2026-09-24 07:09:58 on dist-test-slave-14ww"
server_key: "7fde23b33d77d1e7db38e8a197e68311"
server_key_iv: "5042990a80c3141d5746a50bff682027"
server_key_version: "encryptionkey@0"
I20260924 07:09:58.173137 5516 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 07:09:58.173905 5530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:58.174098 5516 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:09:58.174194 5516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "0d8048bd1e0747d0be77403cda8c3986"
format_stamp: "Formatted at 2026-09-24 07:09:58 on dist-test-slave-14ww"
server_key: "7fde23b33d77d1e7db38e8a197e68311"
server_key_iv: "5042990a80c3141d5746a50bff682027"
server_key_version: "encryptionkey@0"
I20260924 07:09:58.174281 5516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:58.182343 5516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:58.182603 5516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:58.182747 5516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:58.186487 5516 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37597
I20260924 07:09:58.186513 5582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37597 every 8 connection(s)
I20260924 07:09:58.186822 5516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:09:58.187695 5583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:58.189841 5583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d8048bd1e0747d0be77403cda8c3986: Bootstrap starting.
I20260924 07:09:58.190344 5583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d8048bd1e0747d0be77403cda8c3986: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:58.190568 5583 log.cc:824] T 00000000000000000000000000000000 P 0d8048bd1e0747d0be77403cda8c3986: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:58.191149 5583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d8048bd1e0747d0be77403cda8c3986: No bootstrap required, opened a new log
I20260924 07:09:58.192586 5583 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d8048bd1e0747d0be77403cda8c3986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8048bd1e0747d0be77403cda8c3986" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37597 } }
I20260924 07:09:58.192710 5583 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d8048bd1e0747d0be77403cda8c3986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:58.192780 5583 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d8048bd1e0747d0be77403cda8c3986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d8048bd1e0747d0be77403cda8c3986, State: Initialized, Role: FOLLOWER
I20260924 07:09:58.192936 5583 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d8048bd1e0747d0be77403cda8c3986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8048bd1e0747d0be77403cda8c3986" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37597 } }
I20260924 07:09:58.193018 5583 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d8048bd1e0747d0be77403cda8c3986 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:09:58.193040 5583 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d8048bd1e0747d0be77403cda8c3986 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:09:58.193068 5583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d8048bd1e0747d0be77403cda8c3986 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:58.193501 5583 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d8048bd1e0747d0be77403cda8c3986 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8048bd1e0747d0be77403cda8c3986" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37597 } }
I20260924 07:09:58.193578 5583 leader_election.cc:302] T 00000000000000000000000000000000 P 0d8048bd1e0747d0be77403cda8c3986 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d8048bd1e0747d0be77403cda8c3986; no voters:
I20260924 07:09:58.193717 5583 leader_election.cc:288] T 00000000000000000000000000000000 P 0d8048bd1e0747d0be77403cda8c3986 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:09:58.193758 5586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d8048bd1e0747d0be77403cda8c3986 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:09:58.193890 5586 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d8048bd1e0747d0be77403cda8c3986 [term 1 LEADER]: Becoming Leader. State: Replica: 0d8048bd1e0747d0be77403cda8c3986, State: Running, Role: LEADER
I20260924 07:09:58.193964 5583 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d8048bd1e0747d0be77403cda8c3986 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:09:58.193979 5586 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d8048bd1e0747d0be77403cda8c3986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8048bd1e0747d0be77403cda8c3986" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37597 } }
I20260924 07:09:58.194007 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 5516
I20260924 07:09:58.194114 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:09:58.194278 5586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d8048bd1e0747d0be77403cda8c3986 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d8048bd1e0747d0be77403cda8c3986. Latest consensus state: current_term: 1 leader_uuid: "0d8048bd1e0747d0be77403cda8c3986" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8048bd1e0747d0be77403cda8c3986" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37597 } } }
I20260924 07:09:58.194308 392 external_mini_cluster.cc:1428] Setting key 55f40999175dfbcdf112c28bbdcca93b
I20260924 07:09:58.194347 5586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d8048bd1e0747d0be77403cda8c3986 [sys.catalog]: This master's current role is: LEADER
I20260924 07:09:58.194506 5587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d8048bd1e0747d0be77403cda8c3986 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d8048bd1e0747d0be77403cda8c3986" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8048bd1e0747d0be77403cda8c3986" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37597 } } }
I20260924 07:09:58.194613 5591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:09:58.194655 5587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d8048bd1e0747d0be77403cda8c3986 [sys.catalog]: This master's current role is: LEADER
I20260924 07:09:58.195149 5591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:09:58.196852 5591 catalog_manager.cc:1347] Generated new cluster ID: 524cfa8852eb45d4bce7590041e6e979
I20260924 07:09:58.196903 5591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:09:58.200863 5599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:58.194898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36958 (local address 127.0.98.62:37597)
0924 07:09:58.195405 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:58.195414 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:58.195445 (+ 31us) server_negotiation.cc:407] Connection header received
0924 07:09:58.195761 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:58.195769 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:58.195852 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:58.195989 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:58.196286 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:58.197611 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:58.198688 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:58.199327 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:58.199453 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:58.200090 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:58.200093 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:58.200094 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:09:58.200295 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:58.200336 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:09:58.200349 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:58.200419 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:58.200482 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 07:09:58.200640 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":83,"threads_started":1}
I20260924 07:09:58.201694 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:09:58.211941 5591 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:09:58.212421 5591 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:09:58.218645 5591 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d8048bd1e0747d0be77403cda8c3986: Generated new TSK 0
I20260924 07:09:58.218812 5591 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 07:09:58.241597 5613 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 07:09:58.241693 5613 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 07:09:58.241909 5599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:58.226750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36964 (local address 127.0.98.62:37597)
0924 07:09:58.226793 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:58.226795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:58.226917 (+ 122us) server_negotiation.cc:407] Connection header received
0924 07:09:58.226980 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:58.226980 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:58.226992 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:58.227063 (+ 71us) server_negotiation.cc:226] Negotiated authn=JWT
0924 07:09:58.227189 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:58.227936 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:58.228662 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:58.228989 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:58.229059 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:58.229221 (+ 162us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 07:09:58.241711 (+ 12490us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33629/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 07:09:58.241838 (+ 127us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36964: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33629/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":24}
W20260924 07:09:58.241973 5599 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36964: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33629/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 07:09:58.242300 5612 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:09:58.226651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37597 (local address 127.0.0.1:36964)
0924 07:09:58.226852 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:58.226861 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:58.226915 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:58.227074 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:58.227074 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:58.227076 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 07:09:58.227127 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:58.227129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:58.228018 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:58.228019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:58.228582 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:58.228584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:58.229057 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:58.229060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:58.229104 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:58.229119 (+ 15us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 07:09:58.241873 (+ 12754us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33629/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 07:09:58.241985 (+ 112us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:37597: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33629/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":104,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:37597: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33629/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 07:09:58.242750 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 5516
2026-09-24T07:09:58Z chronyd exiting
I20260924 07:09:58.278597 392 test_util.cc:175] -----------------------------------------------
I20260924 07:09:58.278682 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790233772407728-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (208 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 07:09:58.279404 392 mini_oidc.cc:152] Starting JWKS server
I20260924 07:09:58.280061 392 webserver.cc:460] Webserver started at https://127.0.0.1:36127/ using document root <none> and password file <none>
I20260924 07:09:58.280102 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 07:09:58.280253 392 webserver.cc:460] Webserver started at http://127.0.0.1:33093/ using document root <none> and password file <none>
2026-09-24T07:09:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:09:58Z Disabled control of system clock
I20260924 07:09:58.287611 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33581
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33581
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36127/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233772407728-392-0/testchainca.cert with env {}
W20260924 07:09:58.365332 5619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:58.365562 5619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:58.365597 5619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:58.366747 5619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:09:58.366814 5619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:58.366837 5619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:58.366868 5619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:09:58.366892 5619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 07:09:58.366918 5619 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 07:09:58.366946 5619 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36127/jwks.json
I20260924 07:09:58.368404 5619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33581
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36127/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233772407728-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:58.368655 5619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:58.368868 5619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:58.373461 5625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:58.373512 5624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:58.373533 5627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:58.373787 5619 server_base.cc:1034] running on GCE node
I20260924 07:09:58.374058 5619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:58.374334 5619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:58.375517 5619 hybrid_clock.cc:648] HybridClock initialized: now 1790233798375495 us; error 33 us; skew 500 ppm
I20260924 07:09:58.376658 5619 webserver.cc:460] Webserver started at http://127.0.98.62:42611/ using document root <none> and password file <none>
I20260924 07:09:58.376904 5619 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:58.376994 5619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:58.377141 5619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:58.378141 5619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "6bd8bf8f48784d849e1c037d49992300"
format_stamp: "Formatted at 2026-09-24 07:09:58 on dist-test-slave-14ww"
server_key: "01432f82ad84db5aef71afc42392e5ce"
server_key_iv: "f8cb6515fee1ebf3b9e571507e4d54e7"
server_key_version: "encryptionkey@0"
I20260924 07:09:58.378515 5619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "6bd8bf8f48784d849e1c037d49992300"
format_stamp: "Formatted at 2026-09-24 07:09:58 on dist-test-slave-14ww"
server_key: "01432f82ad84db5aef71afc42392e5ce"
server_key_iv: "f8cb6515fee1ebf3b9e571507e4d54e7"
server_key_version: "encryptionkey@0"
I20260924 07:09:58.379822 5619 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:09:58.380581 5633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:58.380825 5619 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:09:58.380906 5619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "6bd8bf8f48784d849e1c037d49992300"
format_stamp: "Formatted at 2026-09-24 07:09:58 on dist-test-slave-14ww"
server_key: "01432f82ad84db5aef71afc42392e5ce"
server_key_iv: "f8cb6515fee1ebf3b9e571507e4d54e7"
server_key_version: "encryptionkey@0"
I20260924 07:09:58.381009 5619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:58.390040 5619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:58.390302 5619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:58.390445 5619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:58.394543 5619 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33581
I20260924 07:09:58.394603 5685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33581 every 8 connection(s)
I20260924 07:09:58.394878 5619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:09:58.395849 5686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:58.398058 5686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bd8bf8f48784d849e1c037d49992300: Bootstrap starting.
I20260924 07:09:58.398597 5686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bd8bf8f48784d849e1c037d49992300: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:58.398821 5686 log.cc:824] T 00000000000000000000000000000000 P 6bd8bf8f48784d849e1c037d49992300: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:58.399513 5686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bd8bf8f48784d849e1c037d49992300: No bootstrap required, opened a new log
I20260924 07:09:58.401131 5686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bd8bf8f48784d849e1c037d49992300 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd8bf8f48784d849e1c037d49992300" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } }
I20260924 07:09:58.401264 5686 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bd8bf8f48784d849e1c037d49992300 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:58.401286 5686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bd8bf8f48784d849e1c037d49992300 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bd8bf8f48784d849e1c037d49992300, State: Initialized, Role: FOLLOWER
I20260924 07:09:58.401472 5686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bd8bf8f48784d849e1c037d49992300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd8bf8f48784d849e1c037d49992300" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } }
I20260924 07:09:58.401506 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 5619
I20260924 07:09:58.401561 5686 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bd8bf8f48784d849e1c037d49992300 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:09:58.401600 5686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bd8bf8f48784d849e1c037d49992300 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:09:58.401607 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:09:58.401651 5686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bd8bf8f48784d849e1c037d49992300 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:58.401798 392 external_mini_cluster.cc:1428] Setting key 2b6905a887aef170c55b85ee09b8cfe4
I20260924 07:09:58.402357 5686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bd8bf8f48784d849e1c037d49992300 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd8bf8f48784d849e1c037d49992300" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } }
I20260924 07:09:58.402488 5686 leader_election.cc:302] T 00000000000000000000000000000000 P 6bd8bf8f48784d849e1c037d49992300 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bd8bf8f48784d849e1c037d49992300; no voters:
I20260924 07:09:58.402663 5686 leader_election.cc:288] T 00000000000000000000000000000000 P 6bd8bf8f48784d849e1c037d49992300 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:09:58.402688 5691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bd8bf8f48784d849e1c037d49992300 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:09:58.402796 5691 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bd8bf8f48784d849e1c037d49992300 [term 1 LEADER]: Becoming Leader. State: Replica: 6bd8bf8f48784d849e1c037d49992300, State: Running, Role: LEADER
I20260924 07:09:58.402877 5691 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bd8bf8f48784d849e1c037d49992300 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd8bf8f48784d849e1c037d49992300" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } }
I20260924 07:09:58.402920 5686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bd8bf8f48784d849e1c037d49992300 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:09:58.403201 5693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bd8bf8f48784d849e1c037d49992300 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bd8bf8f48784d849e1c037d49992300" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd8bf8f48784d849e1c037d49992300" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } } }
I20260924 07:09:58.403358 5693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bd8bf8f48784d849e1c037d49992300 [sys.catalog]: This master's current role is: LEADER
I20260924 07:09:58.403318 5692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bd8bf8f48784d849e1c037d49992300 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bd8bf8f48784d849e1c037d49992300. Latest consensus state: current_term: 1 leader_uuid: "6bd8bf8f48784d849e1c037d49992300" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd8bf8f48784d849e1c037d49992300" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33581 } } }
I20260924 07:09:58.403470 5692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bd8bf8f48784d849e1c037d49992300 [sys.catalog]: This master's current role is: LEADER
W20260924 07:09:58.404721 5707 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6bd8bf8f48784d849e1c037d49992300: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:09:58.404816 5707 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:09:58.404878 5699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:09:58.405061 5699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:09:58.406612 5699 catalog_manager.cc:1347] Generated new cluster ID: 5bc2c18fef5d44d59992ddda0adf2911
I20260924 07:09:58.406665 5699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:09:58.409646 5690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:58.402273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58324 (local address 127.0.98.62:33581)
0924 07:09:58.402476 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:58.402485 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:58.402515 (+ 30us) server_negotiation.cc:407] Connection header received
0924 07:09:58.402630 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:58.402636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:58.402696 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:58.403085 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:58.403187 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:58.404491 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:58.405576 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:58.406130 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:58.406285 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:58.408980 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:58.408984 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:58.408984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:09:58.409162 (+ 178us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:58.409194 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:09:58.409205 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:58.409286 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:58.409360 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 07:09:58.409436 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":102,"threads_started":1}
I20260924 07:09:58.410466 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:09:58.430192 5699 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:09:58.430691 5699 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:09:58.444811 5699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bd8bf8f48784d849e1c037d49992300: Generated new TSK 0
I20260924 07:09:58.444980 5699 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 07:09:58.468986 5716 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 07:09:58.469056 5716 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 07:09:58.469342 5690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:58.455146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58340 (local address 127.0.98.62:33581)
0924 07:09:58.455211 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:58.455213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:58.455354 (+ 141us) server_negotiation.cc:407] Connection header received
0924 07:09:58.455425 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:58.455426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:58.455436 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:58.455495 (+ 59us) server_negotiation.cc:226] Negotiated authn=JWT
0924 07:09:58.455646 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:58.456120 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:58.457285 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:58.457469 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:58.457517 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:58.457700 (+ 183us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 07:09:58.469176 (+ 11476us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36127/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 07:09:58.469296 (+ 120us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58340: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36127/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":40}
W20260924 07:09:58.469405 5690 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58340: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36127/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 07:09:58.469515 5715 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:09:58.455105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33581 (local address 127.0.0.1:58340)
0924 07:09:58.455274 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:58.455285 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:58.455359 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:58.455516 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:58.455516 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:58.455518 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 07:09:58.455583 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:58.455586 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:58.456215 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:58.456216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:58.457196 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:58.457199 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:58.457540 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:58.457543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:58.457604 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:58.457609 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 07:09:58.469295 (+ 11686us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36127/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 07:09:58.469404 (+ 109us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:33581: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36127/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":96,"threads_started":1}
I20260924 07:09:58.469841 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 5619
2026-09-24T07:09:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (227 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 07:09:58.507360 392 mini_oidc.cc:152] Starting JWKS server
I20260924 07:09:58.508046 392 webserver.cc:460] Webserver started at https://127.0.0.1:43873/ using document root <none> and password file <none>
I20260924 07:09:58.508086 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 07:09:58.508275 392 webserver.cc:460] Webserver started at http://127.0.0.1:43269/ using document root <none> and password file <none>
2026-09-24T07:09:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:09:58Z Disabled control of system clock
I20260924 07:09:58.515550 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32869
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32869
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43873/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 07:09:58.596262 5722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:58.596477 5722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:58.596503 5722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:58.597700 5722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:09:58.597739 5722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:58.597752 5722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:58.597764 5722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:09:58.597779 5722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 07:09:58.597790 5722 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 07:09:58.597801 5722 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43873/jwks.json
I20260924 07:09:58.599104 5722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32869
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43873/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:58.599300 5722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:58.599535 5722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:58.604496 5728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:58.604383 5730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:58.604707 5727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:58.604774 5722 server_base.cc:1034] running on GCE node
I20260924 07:09:58.605055 5722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:58.605340 5722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:58.606521 5722 hybrid_clock.cc:648] HybridClock initialized: now 1790233798606505 us; error 45 us; skew 500 ppm
I20260924 07:09:58.607800 5722 webserver.cc:460] Webserver started at http://127.0.98.62:41629/ using document root <none> and password file <none>
I20260924 07:09:58.608058 5722 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:58.608131 5722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:58.608412 5722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:58.609513 5722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "12f4ce3ab4d9450390aea4215af8df3d"
format_stamp: "Formatted at 2026-09-24 07:09:58 on dist-test-slave-14ww"
server_key: "c466701a6e5177759e760a19bbd4ba63"
server_key_iv: "88062fe75aff8238ddda0f1a6081269f"
server_key_version: "encryptionkey@0"
I20260924 07:09:58.609876 5722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "12f4ce3ab4d9450390aea4215af8df3d"
format_stamp: "Formatted at 2026-09-24 07:09:58 on dist-test-slave-14ww"
server_key: "c466701a6e5177759e760a19bbd4ba63"
server_key_iv: "88062fe75aff8238ddda0f1a6081269f"
server_key_version: "encryptionkey@0"
I20260924 07:09:58.611162 5722 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 07:09:58.611841 5736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:58.612010 5722 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:09:58.612107 5722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "12f4ce3ab4d9450390aea4215af8df3d"
format_stamp: "Formatted at 2026-09-24 07:09:58 on dist-test-slave-14ww"
server_key: "c466701a6e5177759e760a19bbd4ba63"
server_key_iv: "88062fe75aff8238ddda0f1a6081269f"
server_key_version: "encryptionkey@0"
I20260924 07:09:58.612187 5722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:58.636539 5722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:58.636924 5722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:58.637085 5722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:58.641114 5722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32869
I20260924 07:09:58.641149 5788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32869 every 8 connection(s)
I20260924 07:09:58.641503 5722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:09:58.642155 5789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:58.644696 5789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12f4ce3ab4d9450390aea4215af8df3d: Bootstrap starting.
I20260924 07:09:58.645290 5789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12f4ce3ab4d9450390aea4215af8df3d: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:58.645583 5789 log.cc:824] T 00000000000000000000000000000000 P 12f4ce3ab4d9450390aea4215af8df3d: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:58.646271 5789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12f4ce3ab4d9450390aea4215af8df3d: No bootstrap required, opened a new log
I20260924 07:09:58.647779 5789 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12f4ce3ab4d9450390aea4215af8df3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f4ce3ab4d9450390aea4215af8df3d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32869 } }
I20260924 07:09:58.647912 5789 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12f4ce3ab4d9450390aea4215af8df3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:58.647934 5789 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12f4ce3ab4d9450390aea4215af8df3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12f4ce3ab4d9450390aea4215af8df3d, State: Initialized, Role: FOLLOWER
I20260924 07:09:58.648098 5789 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12f4ce3ab4d9450390aea4215af8df3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f4ce3ab4d9450390aea4215af8df3d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32869 } }
I20260924 07:09:58.648161 5789 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12f4ce3ab4d9450390aea4215af8df3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:09:58.648182 5789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12f4ce3ab4d9450390aea4215af8df3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:09:58.648208 5789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12f4ce3ab4d9450390aea4215af8df3d [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:58.648725 5789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12f4ce3ab4d9450390aea4215af8df3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f4ce3ab4d9450390aea4215af8df3d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32869 } }
I20260924 07:09:58.648882 5789 leader_election.cc:302] T 00000000000000000000000000000000 P 12f4ce3ab4d9450390aea4215af8df3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12f4ce3ab4d9450390aea4215af8df3d; no voters:
I20260924 07:09:58.649057 5789 leader_election.cc:288] T 00000000000000000000000000000000 P 12f4ce3ab4d9450390aea4215af8df3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:09:58.649117 5792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12f4ce3ab4d9450390aea4215af8df3d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:09:58.649313 5789 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12f4ce3ab4d9450390aea4215af8df3d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:09:58.649322 5792 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12f4ce3ab4d9450390aea4215af8df3d [term 1 LEADER]: Becoming Leader. State: Replica: 12f4ce3ab4d9450390aea4215af8df3d, State: Running, Role: LEADER
I20260924 07:09:58.649434 5792 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12f4ce3ab4d9450390aea4215af8df3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f4ce3ab4d9450390aea4215af8df3d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32869 } }
I20260924 07:09:58.649907 5794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12f4ce3ab4d9450390aea4215af8df3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12f4ce3ab4d9450390aea4215af8df3d. Latest consensus state: current_term: 1 leader_uuid: "12f4ce3ab4d9450390aea4215af8df3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f4ce3ab4d9450390aea4215af8df3d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32869 } } }
I20260924 07:09:58.649991 5794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12f4ce3ab4d9450390aea4215af8df3d [sys.catalog]: This master's current role is: LEADER
I20260924 07:09:58.650032 5793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12f4ce3ab4d9450390aea4215af8df3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12f4ce3ab4d9450390aea4215af8df3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12f4ce3ab4d9450390aea4215af8df3d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32869 } } }
I20260924 07:09:58.650079 5793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12f4ce3ab4d9450390aea4215af8df3d [sys.catalog]: This master's current role is: LEADER
I20260924 07:09:58.650389 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 5722
I20260924 07:09:58.650498 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:09:58.650728 392 external_mini_cluster.cc:1428] Setting key ee4c5a30447b5d5fb45c203391fe9049
I20260924 07:09:58.651010 5801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:09:58.651522 5801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:09:58.653177 5801 catalog_manager.cc:1347] Generated new cluster ID: d55ed76906d94682ac095ffe01befa06
I20260924 07:09:58.653228 5801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:09:58.656261 5809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:58.651179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43874 (local address 127.0.98.62:32869)
0924 07:09:58.651650 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:58.651659 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:58.651741 (+ 82us) server_negotiation.cc:407] Connection header received
0924 07:09:58.651852 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:58.651859 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:58.651935 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:58.652086 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:58.652312 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:58.653553 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:58.654322 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:58.654750 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:58.654876 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:58.655612 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:58.655616 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:58.655616 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:09:58.655791 (+ 175us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:58.655821 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:09:58.655832 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:58.655912 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:58.655983 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 07:09:58.656061 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":61,"threads_started":1}
I20260924 07:09:58.657176 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:09:58.668846 5801 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:09:58.669353 5801 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:09:58.673224 5801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12f4ce3ab4d9450390aea4215af8df3d: Generated new TSK 0
I20260924 07:09:58.673384 5801 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:09:58.690284 5809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:58.686557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43884 (local address 127.0.98.62:32869)
0924 07:09:58.686599 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:58.686601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:58.686724 (+ 123us) server_negotiation.cc:407] Connection header received
0924 07:09:58.686776 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:58.686777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:58.686806 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:58.686879 (+ 73us) server_negotiation.cc:226] Negotiated authn=JWT
0924 07:09:58.687013 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:58.687767 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:58.688587 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:58.688910 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:58.688986 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:58.689174 (+ 188us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 07:09:58.690072 (+ 898us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43873/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 07:09:58.690209 (+ 137us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43884: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43873/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 07:09:58.690340 5809 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43884: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43873/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 07:09:58.692250 5818 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:09:58.686512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32869 (local address 127.0.0.1:43884)
0924 07:09:58.686649 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:58.686658 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:58.686718 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:58.686882 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:58.686882 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:58.686884 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 07:09:58.686946 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:58.686948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:58.687867 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:58.687868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:58.688504 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:58.688506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:58.689009 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:58.689016 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:58.689062 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:58.689066 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 07:09:58.690183 (+ 1117us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43873/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 07:09:58.692161 (+ 1978us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:32869: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43873/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":1793,"thread_start_us":73,"threads_started":1}
I20260924 07:09:58.692443 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 5722
E20260924 07:09:58.700947 5819 webserver.cc:533] Webserver: sslize failed:
E20260924 07:09:58.701035 5819 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-24T07:09:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (221 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T07:09:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:09:58Z Disabled control of system clock
I20260924 07:09:58.735900 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35515
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35515
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233772407728-392-0/testchainca.cert with env {}
W20260924 07:09:58.814492 5823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:09:58.814693 5823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:09:58.814714 5823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:09:58.815807 5823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:09:58.815842 5823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:09:58.815855 5823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:09:58.815867 5823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:09:58.815878 5823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 07:09:58.815891 5823 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 07:09:58.815901 5823 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 07:09:58.817348 5823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35515
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233772407728-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:09:58.817538 5823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:09:58.817687 5823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:09:58.822402 5828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:58.822480 5829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:09:58.822443 5831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:09:58.822602 5823 server_base.cc:1034] running on GCE node
I20260924 07:09:58.822849 5823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:09:58.823167 5823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:09:58.824376 5823 hybrid_clock.cc:648] HybridClock initialized: now 1790233798824359 us; error 39 us; skew 500 ppm
I20260924 07:09:58.825793 5823 webserver.cc:460] Webserver started at http://127.0.98.62:35311/ using document root <none> and password file <none>
I20260924 07:09:58.826102 5823 fs_manager.cc:359] Metadata directory not provided
I20260924 07:09:58.826171 5823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:09:58.826314 5823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:09:58.827425 5823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "10de7557dac2427ab9e0fbdde7979b78"
format_stamp: "Formatted at 2026-09-24 07:09:58 on dist-test-slave-14ww"
server_key: "71d966cec15c99029158ac3f0f0057e7"
server_key_iv: "2eb126e376a2cc88c3b5c178d6185ea1"
server_key_version: "encryptionkey@0"
I20260924 07:09:58.827821 5823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "10de7557dac2427ab9e0fbdde7979b78"
format_stamp: "Formatted at 2026-09-24 07:09:58 on dist-test-slave-14ww"
server_key: "71d966cec15c99029158ac3f0f0057e7"
server_key_iv: "2eb126e376a2cc88c3b5c178d6185ea1"
server_key_version: "encryptionkey@0"
I20260924 07:09:58.829212 5823 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:09:58.830006 5837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:09:58.830180 5823 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:09:58.830299 5823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "10de7557dac2427ab9e0fbdde7979b78"
format_stamp: "Formatted at 2026-09-24 07:09:58 on dist-test-slave-14ww"
server_key: "71d966cec15c99029158ac3f0f0057e7"
server_key_iv: "2eb126e376a2cc88c3b5c178d6185ea1"
server_key_version: "encryptionkey@0"
I20260924 07:09:58.830381 5823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:09:58.844125 5823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:09:58.844404 5823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:09:58.844559 5823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:09:58.848263 5823 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35515
I20260924 07:09:58.848289 5889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35515 every 8 connection(s)
I20260924 07:09:58.848609 5823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:09:58.849507 5890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:09:58.849745 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 5823
I20260924 07:09:58.849856 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:09:58.850034 392 external_mini_cluster.cc:1428] Setting key 5bf34ce4eb76b328bb728615252a7dcd
I20260924 07:09:58.852095 5890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10de7557dac2427ab9e0fbdde7979b78: Bootstrap starting.
I20260924 07:09:58.852763 5890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10de7557dac2427ab9e0fbdde7979b78: Neither blocks nor log segments found. Creating new log.
I20260924 07:09:58.853062 5890 log.cc:824] T 00000000000000000000000000000000 P 10de7557dac2427ab9e0fbdde7979b78: Log is configured to *not* fsync() on all Append() calls
I20260924 07:09:58.853746 5890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10de7557dac2427ab9e0fbdde7979b78: No bootstrap required, opened a new log
I20260924 07:09:58.855424 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:58.850565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45072 (local address 127.0.98.62:35515)
0924 07:09:58.850742 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:58.850749 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:58.850772 (+ 23us) server_negotiation.cc:407] Connection header received
0924 07:09:58.851171 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:58.851177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:58.851240 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:58.851352 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:09:58.851588 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:58.852495 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:58.853556 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:58.854004 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:58.854093 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:58.854782 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:09:58.854786 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:09:58.854786 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:09:58.854943 (+ 157us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:09:58.854973 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:09:58.854984 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:09:58.855046 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 07:09:58.855138 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 07:09:58.855232 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":76,"threads_started":1}
I20260924 07:09:58.856009 5890 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10de7557dac2427ab9e0fbdde7979b78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10de7557dac2427ab9e0fbdde7979b78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35515 } }
I20260924 07:09:58.856166 5890 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10de7557dac2427ab9e0fbdde7979b78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:09:58.856226 5890 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10de7557dac2427ab9e0fbdde7979b78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10de7557dac2427ab9e0fbdde7979b78, State: Initialized, Role: FOLLOWER
I20260924 07:09:58.856349 5890 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10de7557dac2427ab9e0fbdde7979b78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10de7557dac2427ab9e0fbdde7979b78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35515 } }
I20260924 07:09:58.856401 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:09:58.856423 5890 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10de7557dac2427ab9e0fbdde7979b78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:09:58.856453 5890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10de7557dac2427ab9e0fbdde7979b78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:09:58.856496 5890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10de7557dac2427ab9e0fbdde7979b78 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:09:58.857162 5890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10de7557dac2427ab9e0fbdde7979b78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10de7557dac2427ab9e0fbdde7979b78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35515 } }
I20260924 07:09:58.857285 5890 leader_election.cc:302] T 00000000000000000000000000000000 P 10de7557dac2427ab9e0fbdde7979b78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10de7557dac2427ab9e0fbdde7979b78; no voters:
I20260924 07:09:58.857427 5890 leader_election.cc:288] T 00000000000000000000000000000000 P 10de7557dac2427ab9e0fbdde7979b78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:09:58.857472 5895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10de7557dac2427ab9e0fbdde7979b78 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:09:58.857658 5895 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10de7557dac2427ab9e0fbdde7979b78 [term 1 LEADER]: Becoming Leader. State: Replica: 10de7557dac2427ab9e0fbdde7979b78, State: Running, Role: LEADER
I20260924 07:09:58.857682 5890 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10de7557dac2427ab9e0fbdde7979b78 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:09:58.857743 5895 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10de7557dac2427ab9e0fbdde7979b78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10de7557dac2427ab9e0fbdde7979b78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35515 } }
I20260924 07:09:58.858168 5897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10de7557dac2427ab9e0fbdde7979b78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10de7557dac2427ab9e0fbdde7979b78. Latest consensus state: current_term: 1 leader_uuid: "10de7557dac2427ab9e0fbdde7979b78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10de7557dac2427ab9e0fbdde7979b78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35515 } } }
I20260924 07:09:58.858273 5897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10de7557dac2427ab9e0fbdde7979b78 [sys.catalog]: This master's current role is: LEADER
I20260924 07:09:58.858180 5896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10de7557dac2427ab9e0fbdde7979b78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10de7557dac2427ab9e0fbdde7979b78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10de7557dac2427ab9e0fbdde7979b78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35515 } } }
I20260924 07:09:58.858498 5896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10de7557dac2427ab9e0fbdde7979b78 [sys.catalog]: This master's current role is: LEADER
I20260924 07:09:58.858472 5904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:09:58.859084 5904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:09:58.860539 5904 catalog_manager.cc:1347] Generated new cluster ID: 7071ff5e3c61475bbab5ff67d8a5e5c1
I20260924 07:09:58.860601 5904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:09:58.870833 5904 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:09:58.871343 5904 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:09:58.875758 5904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10de7557dac2427ab9e0fbdde7979b78: Generated new TSK 0
I20260924 07:09:58.875938 5904 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:09:58.953112 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:09:58.888243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45076 (local address 127.0.98.62:35515)
0924 07:09:58.888312 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 07:09:58.888314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:09:58.888423 (+ 109us) server_negotiation.cc:407] Connection header received
0924 07:09:58.888500 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:09:58.888502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:09:58.888520 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:09:58.888599 (+ 79us) server_negotiation.cc:226] Negotiated authn=JWT
0924 07:09:58.888727 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:58.889255 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:58.890027 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:09:58.890223 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:09:58.890308 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:58.890505 (+ 197us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 07:09:58.952882 (+ 62377us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 07:09:58.953053 (+ 171us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45076: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":43}
W20260924 07:09:58.953202 5893 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45076: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 07:09:58.953306 5919 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:09:58.888176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35515 (local address 127.0.0.1:45076)
0924 07:09:58.888351 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 07:09:58.888360 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:09:58.888440 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:09:58.888598 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:09:58.888599 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:09:58.888601 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 07:09:58.888659 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:58.888661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:58.889339 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:58.889340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:58.889934 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:09:58.889935 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:09:58.890316 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:09:58.890322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:09:58.890378 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:09:58.890389 (+ 11us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 07:09:58.953037 (+ 62648us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 07:09:58.953199 (+ 162us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:35515: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":104,"threads_started":1}
I20260924 07:09:58.953596 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 5823
2026-09-24T07:09:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (241 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (86 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (85 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:09:59 dist-test-slave-14ww krb5kdc[5931](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:09:59 dist-test-slave-14ww krb5kdc[5931](info): set up 2 sockets
Sep 24 07:09:59 dist-test-slave-14ww krb5kdc[5931](info): commencing operation
krb5kdc: starting...
W20260924 07:10:00.164887 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:10:00 dist-test-slave-14ww krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:10:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:10:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:10:00.288949 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40827
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40827
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:00.367223 5944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:00.367406 5944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:00.367427 5944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:00.368633 5944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:10:00.368673 5944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:00.368688 5944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:00.368701 5944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:10:00.368762 5944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:10:00.370232 5944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:10:00.370469 5944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:00.370679 5944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:00.375371 5949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:00.375365 5952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:00.375376 5950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:00.376151 5944 server_base.cc:1034] running on GCE node
I20260924 07:10:00.376406 5944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:00.376653 5944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:00.377844 5944 hybrid_clock.cc:648] HybridClock initialized: now 1790233800377815 us; error 43 us; skew 500 ppm
Sep 24 07:10:00 dist-test-slave-14ww krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:00.389002 5944 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:10:00.389380 5944 webserver.cc:460] Webserver started at http://127.0.98.62:41001/ using document root <none> and password file <none>
I20260924 07:10:00.389554 5944 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:00.389595 5944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:00.389689 5944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:00.390652 5944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "fef8142dd1bb480cbc9126776e256caa"
format_stamp: "Formatted at 2026-09-24 07:10:00 on dist-test-slave-14ww"
server_key: "9bc0447092287ebc7bb0b9a9dc881976"
server_key_iv: "7cabc65ae62158500c0ffaca0a9a2eb2"
server_key_version: "encryptionkey@0"
I20260924 07:10:00.390954 5944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "fef8142dd1bb480cbc9126776e256caa"
format_stamp: "Formatted at 2026-09-24 07:10:00 on dist-test-slave-14ww"
server_key: "9bc0447092287ebc7bb0b9a9dc881976"
server_key_iv: "7cabc65ae62158500c0ffaca0a9a2eb2"
server_key_version: "encryptionkey@0"
I20260924 07:10:00.392198 5944 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:10:00.392915 5959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:00.393100 5944 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:10:00.393165 5944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "fef8142dd1bb480cbc9126776e256caa"
format_stamp: "Formatted at 2026-09-24 07:10:00 on dist-test-slave-14ww"
server_key: "9bc0447092287ebc7bb0b9a9dc881976"
server_key_iv: "7cabc65ae62158500c0ffaca0a9a2eb2"
server_key_version: "encryptionkey@0"
I20260924 07:10:00.393214 5944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:00.414573 5944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:00.416826 5944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:00.416931 5944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:00.420518 5944 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40827
I20260924 07:10:00.420576 6011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40827 every 8 connection(s)
I20260924 07:10:00.421036 5944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:10:00.421695 6012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:00.423480 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 5944
I20260924 07:10:00.423573 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:10:00.423810 392 external_mini_cluster.cc:1428] Setting key b1ea6e5ab8025496519a9383f6a2335c
I20260924 07:10:00.423998 6012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fef8142dd1bb480cbc9126776e256caa: Bootstrap starting.
I20260924 07:10:00.424646 6012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fef8142dd1bb480cbc9126776e256caa: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:00.424981 6012 log.cc:824] T 00000000000000000000000000000000 P fef8142dd1bb480cbc9126776e256caa: Log is configured to *not* fsync() on all Append() calls
I20260924 07:10:00.425630 6012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fef8142dd1bb480cbc9126776e256caa: No bootstrap required, opened a new log
I20260924 07:10:00.427374 6012 raft_consensus.cc:348] T 00000000000000000000000000000000 P fef8142dd1bb480cbc9126776e256caa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef8142dd1bb480cbc9126776e256caa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40827 } }
I20260924 07:10:00.427532 6012 raft_consensus.cc:374] T 00000000000000000000000000000000 P fef8142dd1bb480cbc9126776e256caa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:00.427572 6012 raft_consensus.cc:729] T 00000000000000000000000000000000 P fef8142dd1bb480cbc9126776e256caa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef8142dd1bb480cbc9126776e256caa, State: Initialized, Role: FOLLOWER
I20260924 07:10:00.427721 6012 consensus_queue.cc:260] T 00000000000000000000000000000000 P fef8142dd1bb480cbc9126776e256caa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef8142dd1bb480cbc9126776e256caa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40827 } }
I20260924 07:10:00.427832 6012 raft_consensus.cc:388] T 00000000000000000000000000000000 P fef8142dd1bb480cbc9126776e256caa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:10:00.427874 6012 raft_consensus.cc:482] T 00000000000000000000000000000000 P fef8142dd1bb480cbc9126776e256caa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:10:00.427922 6012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fef8142dd1bb480cbc9126776e256caa [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:00.428574 6012 raft_consensus.cc:504] T 00000000000000000000000000000000 P fef8142dd1bb480cbc9126776e256caa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef8142dd1bb480cbc9126776e256caa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40827 } }
I20260924 07:10:00.428700 6012 leader_election.cc:302] T 00000000000000000000000000000000 P fef8142dd1bb480cbc9126776e256caa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fef8142dd1bb480cbc9126776e256caa; no voters:
I20260924 07:10:00.428880 6012 leader_election.cc:288] T 00000000000000000000000000000000 P fef8142dd1bb480cbc9126776e256caa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:10:00.428915 6017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fef8142dd1bb480cbc9126776e256caa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:10:00.429068 6017 raft_consensus.cc:686] T 00000000000000000000000000000000 P fef8142dd1bb480cbc9126776e256caa [term 1 LEADER]: Becoming Leader. State: Replica: fef8142dd1bb480cbc9126776e256caa, State: Running, Role: LEADER
I20260924 07:10:00.429122 6012 sys_catalog.cc:564] T 00000000000000000000000000000000 P fef8142dd1bb480cbc9126776e256caa [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:10:00.429186 6017 consensus_queue.cc:237] T 00000000000000000000000000000000 P fef8142dd1bb480cbc9126776e256caa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef8142dd1bb480cbc9126776e256caa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40827 } }
I20260924 07:10:00.429504 6017 sys_catalog.cc:455] T 00000000000000000000000000000000 P fef8142dd1bb480cbc9126776e256caa [sys.catalog]: SysCatalogTable state changed. Reason: New leader fef8142dd1bb480cbc9126776e256caa. Latest consensus state: current_term: 1 leader_uuid: "fef8142dd1bb480cbc9126776e256caa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef8142dd1bb480cbc9126776e256caa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40827 } } }
I20260924 07:10:00.429590 6017 sys_catalog.cc:458] T 00000000000000000000000000000000 P fef8142dd1bb480cbc9126776e256caa [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:00.429519 6018 sys_catalog.cc:455] T 00000000000000000000000000000000 P fef8142dd1bb480cbc9126776e256caa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fef8142dd1bb480cbc9126776e256caa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fef8142dd1bb480cbc9126776e256caa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40827 } } }
I20260924 07:10:00.429744 6018 sys_catalog.cc:458] T 00000000000000000000000000000000 P fef8142dd1bb480cbc9126776e256caa [sys.catalog]: This master's current role is: LEADER
Sep 24 07:10:00 dist-test-slave-14ww krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:10:00.430614 6024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:10:00.431180 6024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:10:00.432632 6024 catalog_manager.cc:1347] Generated new cluster ID: 9f777a495d374271b7f5ae0dcc4deec6
I20260924 07:10:00.432688 6024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:10:00.434638 6015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:00.424249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59536 (local address 127.0.98.62:40827)
0924 07:10:00.424413 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:00.424423 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:00.424457 (+ 34us) server_negotiation.cc:407] Connection header received
0924 07:10:00.424951 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:00.424957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:00.425002 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:00.425151 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:00.425756 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:00.426608 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:00.427357 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:00.427829 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:00.427946 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:00.430701 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:00.430705 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:00.430705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:00.430717 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:00.432788 (+ 2071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:00.433170 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:00.433173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:00.433173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:00.433264 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:00.433464 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:00.433465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:00.433465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:00.433752 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:00.433887 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:00.434299 (+ 412us) server_negotiation.cc:299] Negotiation successful
0924 07:10:00.434427 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
I20260924 07:10:00.443878 6024 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:10:00.444533 6024 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:10:00.450253 6024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fef8142dd1bb480cbc9126776e256caa: Generated new TSK 0
I20260924 07:10:00.450429 6024 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 07:10:00.477648 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40827
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:00.556305 6038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:00.556488 6038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:00.556512 6038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:00.557657 6038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:00.557703 6038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:00.557729 6038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:10:00.559199 6038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:10:00.559429 6038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:00.559648 6038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:00.564358 6043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:00.564363 6046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:00.564378 6038 server_base.cc:1034] running on GCE node
W20260924 07:10:00.564391 6044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:00.564822 6038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:00.565063 6038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:00.566221 6038 hybrid_clock.cc:648] HybridClock initialized: now 1790233800566203 us; error 31 us; skew 500 ppm
Sep 24 07:10:00 dist-test-slave-14ww krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:00.568571 6038 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:10:00.569070 6038 webserver.cc:460] Webserver started at http://127.0.98.1:45811/ using document root <none> and password file <none>
I20260924 07:10:00.569283 6038 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:00.569350 6038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:00.569504 6038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:00.570487 6038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-0/data/instance:
uuid: "acbe1295c2854d7cbef15b9bc869fc93"
format_stamp: "Formatted at 2026-09-24 07:10:00 on dist-test-slave-14ww"
server_key: "80efe2435c3a563f3d2de277463cfb94"
server_key_iv: "4127ac9aad33a6cea9d4e66512444eb7"
server_key_version: "encryptionkey@0"
I20260924 07:10:00.570849 6038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-0/wal/instance:
uuid: "acbe1295c2854d7cbef15b9bc869fc93"
format_stamp: "Formatted at 2026-09-24 07:10:00 on dist-test-slave-14ww"
server_key: "80efe2435c3a563f3d2de277463cfb94"
server_key_iv: "4127ac9aad33a6cea9d4e66512444eb7"
server_key_version: "encryptionkey@0"
I20260924 07:10:00.572124 6038 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:10:00.572885 6053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:00.573060 6038 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:10:00.573169 6038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-0/wal
uuid: "acbe1295c2854d7cbef15b9bc869fc93"
format_stamp: "Formatted at 2026-09-24 07:10:00 on dist-test-slave-14ww"
server_key: "80efe2435c3a563f3d2de277463cfb94"
server_key_iv: "4127ac9aad33a6cea9d4e66512444eb7"
server_key_version: "encryptionkey@0"
I20260924 07:10:00.573239 6038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:00.587200 6038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:00.589396 6038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:00.589551 6038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:00.589768 6038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:10:00.590054 6038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:10:00.590107 6038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:00.590147 6038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:10:00.590176 6038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:00.596062 6038 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36203
I20260924 07:10:00.596144 6166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36203 every 8 connection(s)
I20260924 07:10:00.596439 6038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 07:10:00 dist-test-slave-14ww krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:10:00.602099 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 6038
I20260924 07:10:00.602223 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:10:00.602413 392 external_mini_cluster.cc:1428] Setting key aac5c86976107c151707c85d6c16d1be
I20260924 07:10:00.605552 6015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:00.597127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42059 (local address 127.0.98.62:40827)
0924 07:10:00.597168 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:00.597170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:00.597353 (+ 183us) server_negotiation.cc:407] Connection header received
0924 07:10:00.598311 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:00.598312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:00.598323 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:00.598380 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:00.598942 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:00.599436 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:00.600365 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:00.600552 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:00.600600 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:00.602709 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:00.602710 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:00.602711 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:00.602717 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:00.604197 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:00.604612 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:00.604613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:00.604613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:00.604672 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:00.604942 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:00.604943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:00.604943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:00.605052 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:00.605129 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:00.605381 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 07:10:00.605463 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 07:10:00.605726 6169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:00.597092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40827 (local address 127.0.98.1:42059)
0924 07:10:00.597239 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:00.597277 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:00.598153 (+ 876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:00.598455 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:00.598459 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:00.598702 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:00.598871 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:00.598874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:00.599508 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:00.599508 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:00.600297 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:00.600299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:00.600637 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:00.600638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:00.600748 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:00.601157 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:00.601165 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:00.602528 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:00.604298 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:00.604304 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:00.604306 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:00.604516 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:00.604771 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:00.604771 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:00.604772 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:00.604876 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:00.605122 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:00.605127 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:00.605257 (+ 130us) client_negotiation.cc:769] Sending connection context
0924 07:10:00.605360 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 07:10:00.605500 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":83,"threads_started":1}
I20260924 07:10:00.606254 6167 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40827
I20260924 07:10:00.606353 6167 heartbeater.cc:464] Registering TS with master...
I20260924 07:10:00.606530 6167 heartbeater.cc:511] Master 127.0.98.62:40827 requested a full tablet report, sending...
I20260924 07:10:00.607059 5976 ts_manager.cc:194] Registered new tserver with Master: acbe1295c2854d7cbef15b9bc869fc93 (127.0.98.1:36203)
I20260924 07:10:00.608028 5976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42059
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 07:10:00.627170 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40827
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:00.705194 6172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:00.705420 6172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:00.705456 6172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:00.706573 6172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:00.706643 6172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:00.706697 6172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:10:00.708148 6172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:10:00.708403 6172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:00.708624 6172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:00.713513 6177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:00.713574 6178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:00.713523 6180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:00.713853 6172 server_base.cc:1034] running on GCE node
I20260924 07:10:00.714120 6172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:00.714393 6172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:00.715547 6172 hybrid_clock.cc:648] HybridClock initialized: now 1790233800715532 us; error 34 us; skew 500 ppm
Sep 24 07:10:00 dist-test-slave-14ww krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:00.718011 6172 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:10:00.718355 6172 webserver.cc:460] Webserver started at http://127.0.98.2:43507/ using document root <none> and password file <none>
I20260924 07:10:00.718546 6172 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:00.718585 6172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:00.718673 6172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:00.719578 6172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-1/data/instance:
uuid: "d6153904e82146f1b7c0f4afe10de770"
format_stamp: "Formatted at 2026-09-24 07:10:00 on dist-test-slave-14ww"
server_key: "e84ecdfe7946a0235683d4412a5daace"
server_key_iv: "3973d49d85819ad3c7d81e86e6a5e1e6"
server_key_version: "encryptionkey@0"
I20260924 07:10:00.719861 6172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d6153904e82146f1b7c0f4afe10de770"
format_stamp: "Formatted at 2026-09-24 07:10:00 on dist-test-slave-14ww"
server_key: "e84ecdfe7946a0235683d4412a5daace"
server_key_iv: "3973d49d85819ad3c7d81e86e6a5e1e6"
server_key_version: "encryptionkey@0"
I20260924 07:10:00.721081 6172 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 07:10:00.721745 6187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:00.721896 6172 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:10:00.721975 6172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-1/wal
uuid: "d6153904e82146f1b7c0f4afe10de770"
format_stamp: "Formatted at 2026-09-24 07:10:00 on dist-test-slave-14ww"
server_key: "e84ecdfe7946a0235683d4412a5daace"
server_key_iv: "3973d49d85819ad3c7d81e86e6a5e1e6"
server_key_version: "encryptionkey@0"
I20260924 07:10:00.722026 6172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:00.754313 6172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:00.756100 6172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:00.756202 6172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:00.756371 6172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:10:00.756683 6172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:10:00.756716 6172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:00.756762 6172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:10:00.756775 6172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:00.762768 6172 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42779
I20260924 07:10:00.762830 6300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42779 every 8 connection(s)
I20260924 07:10:00.763135 6172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 07:10:00 dist-test-slave-14ww krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:10:00.771664 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 6172
I20260924 07:10:00.771788 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:10:00.771922 6015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:00.763849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38953 (local address 127.0.98.62:40827)
0924 07:10:00.763890 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:00.763892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:00.764119 (+ 227us) server_negotiation.cc:407] Connection header received
0924 07:10:00.765156 (+ 1037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:00.765157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:00.765169 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:00.765232 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:00.765836 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:00.766337 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:00.767330 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:00.767522 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:00.767633 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:00.769578 (+ 1945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:00.769579 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:00.769579 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:00.769585 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:00.770763 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:00.771102 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:00.771103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:00.771103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:00.771152 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:00.771350 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:00.771350 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:00.771350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:00.771441 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:00.771490 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:00.771749 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 07:10:00.771831 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 07:10:00.772025 392 external_mini_cluster.cc:1428] Setting key c264e7d4536c8a097ca9fe6b007780e4
I20260924 07:10:00.772123 6303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:00.763814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40827 (local address 127.0.98.2:38953)
0924 07:10:00.763990 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:00.764041 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:00.765029 (+ 988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:00.765302 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:00.765306 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:00.765555 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:00.765733 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:00.765737 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:00.766428 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:00.766428 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:00.767220 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:00.767222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:00.767638 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:00.767639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:00.767757 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:00.768172 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:00.768180 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:00.769499 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:00.770831 (+ 1332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:00.770836 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:00.770838 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:00.771038 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:00.771209 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:00.771210 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:00.771210 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:00.771296 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:00.771494 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:00.771497 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:00.771599 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 07:10:00.771720 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 07:10:00.771868 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":93,"threads_started":1}
I20260924 07:10:00.772542 6301 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40827
I20260924 07:10:00.772634 6301 heartbeater.cc:464] Registering TS with master...
I20260924 07:10:00.772833 6301 heartbeater.cc:511] Master 127.0.98.62:40827 requested a full tablet report, sending...
I20260924 07:10:00.773242 5975 ts_manager.cc:194] Registered new tserver with Master: d6153904e82146f1b7c0f4afe10de770 (127.0.98.2:42779)
I20260924 07:10:00.773623 5975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38953
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 07:10:00.795352 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40827
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:00.874131 6306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:00.874317 6306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:00.874339 6306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:00.875458 6306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:00.875500 6306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:00.875525 6306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:10:00.877019 6306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:10:00.877252 6306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:00.877489 6306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:00.882481 6311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:00.882488 6312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:00.882598 6314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:00.882822 6306 server_base.cc:1034] running on GCE node
I20260924 07:10:00.883038 6306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:00.883297 6306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:00.884467 6306 hybrid_clock.cc:648] HybridClock initialized: now 1790233800884455 us; error 33 us; skew 500 ppm
Sep 24 07:10:00 dist-test-slave-14ww krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:00.887214 6306 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:10:00.887640 6306 webserver.cc:460] Webserver started at http://127.0.98.3:37373/ using document root <none> and password file <none>
I20260924 07:10:00.887861 6306 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:00.887904 6306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:00.888065 6306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:00.889395 6306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-2/data/instance:
uuid: "6e84cc15e5ae4b7981c46412e5489c37"
format_stamp: "Formatted at 2026-09-24 07:10:00 on dist-test-slave-14ww"
server_key: "3b1dfde5ee75408c93dc33c2d25ea11d"
server_key_iv: "fe201dc9d944260377e94c6b93be73fa"
server_key_version: "encryptionkey@0"
I20260924 07:10:00.889814 6306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6e84cc15e5ae4b7981c46412e5489c37"
format_stamp: "Formatted at 2026-09-24 07:10:00 on dist-test-slave-14ww"
server_key: "3b1dfde5ee75408c93dc33c2d25ea11d"
server_key_iv: "fe201dc9d944260377e94c6b93be73fa"
server_key_version: "encryptionkey@0"
I20260924 07:10:00.891216 6306 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:10:00.891927 6321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:00.892097 6306 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:10:00.892198 6306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-2/wal
uuid: "6e84cc15e5ae4b7981c46412e5489c37"
format_stamp: "Formatted at 2026-09-24 07:10:00 on dist-test-slave-14ww"
server_key: "3b1dfde5ee75408c93dc33c2d25ea11d"
server_key_iv: "fe201dc9d944260377e94c6b93be73fa"
server_key_version: "encryptionkey@0"
I20260924 07:10:00.892280 6306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:00.908396 6306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:00.910595 6306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:00.910754 6306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:00.910991 6306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:10:00.911298 6306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:10:00.911350 6306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:00.911391 6306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:10:00.911417 6306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:00.917230 6306 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33621
I20260924 07:10:00.917270 6434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33621 every 8 connection(s)
I20260924 07:10:00.917569 6306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:10:00.919674 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 6306
I20260924 07:10:00.919790 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:10:00.919972 392 external_mini_cluster.cc:1428] Setting key 1137d7cfc45f6aa6b9f619e8f8748b37
Sep 24 07:10:00 dist-test-slave-14ww krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:10:00.927280 6015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:00.918333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51185 (local address 127.0.98.62:40827)
0924 07:10:00.918397 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:00.918399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:00.918610 (+ 211us) server_negotiation.cc:407] Connection header received
0924 07:10:00.919584 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:00.919585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:00.919598 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:00.919651 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:00.920404 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:00.920995 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:00.921995 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:00.922193 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:00.922285 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:00.924458 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:00.924459 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:00.924460 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:00.924467 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:00.925952 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:00.926348 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:00.926349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:00.926350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:00.926409 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:00.926658 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:00.926659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:00.926659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:00.926773 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:00.926850 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:00.927109 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 07:10:00.927184 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 07:10:00.927470 6437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:00.918319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40827 (local address 127.0.98.3:51185)
0924 07:10:00.918477 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:00.918511 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:00.919470 (+ 959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:00.919722 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:00.919726 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:00.920081 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:00.920308 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:00.920312 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:00.921085 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:00.921087 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:00.921900 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:00.921902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:00.922307 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:00.922308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:00.922444 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:00.922977 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:00.922985 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:00.924297 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:00.926044 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:00.926050 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:00.926052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:00.926271 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:00.926476 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:00.926477 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:00.926478 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:00.926584 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:00.926840 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:00.926846 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:00.926981 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 07:10:00.927088 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 07:10:00.927248 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
I20260924 07:10:00.927886 6435 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40827
I20260924 07:10:00.927961 6435 heartbeater.cc:464] Registering TS with master...
I20260924 07:10:00.928151 6435 heartbeater.cc:511] Master 127.0.98.62:40827 requested a full tablet report, sending...
I20260924 07:10:00.928514 5975 ts_manager.cc:194] Registered new tserver with Master: 6e84cc15e5ae4b7981c46412e5489c37 (127.0.98.3:33621)
I20260924 07:10:00.928958 5975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51185
I20260924 07:10:00.931505 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/krb5kdc/test-admin.keytab.
Sep 24 07:10:00 dist-test-slave-14ww krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:00.945983 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 07:10:00.947499 6442 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:00.946767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35436 (local address 127.0.98.1:36203)
0924 07:10:00.946940 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:00.946944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:00.946954 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:10:00.946977 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:00.946979 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:00.946992 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:00.947058 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:00.947437 (+ 379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35436: BlockingRecv error: recv got EOF from 127.0.0.1:35436 (error 108)
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":98,"threads_started":1}
W20260924 07:10:00.947705 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 07:10:00 dist-test-slave-14ww krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:00.948726 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 07:10:00 dist-test-slave-14ww krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:00.949716 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 07:10:00 dist-test-slave-14ww krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 07:10:00.957425 6442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:00.950255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35444 (local address 127.0.98.1:36203)
0924 07:10:00.950324 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:00.950327 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:00.950389 (+ 62us) server_negotiation.cc:407] Connection header received
0924 07:10:00.950433 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:00.950434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:00.950443 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:00.950489 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:00.950922 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:00.951632 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:00.952353 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:00.952658 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:00.952713 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:00.954776 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:00.954783 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:00.954783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:00.954788 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:00.956307 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:00.956626 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:00.956629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:00.956629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:00.956679 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:00.956892 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:00.956893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:00.956893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:00.957009 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:00.957114 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:00.957201 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 07:10:00.957260 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260924 07:10:00.959161 6442 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:00.958607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35456 (local address 127.0.98.1:36203)
0924 07:10:00.958643 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:00.958646 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:00.958691 (+ 45us) server_negotiation.cc:407] Connection header received
0924 07:10:00.958759 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:00.958760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:00.958770 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:00.958839 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:00.959112 (+ 273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35456: BlockingRecv error: recv got EOF from 127.0.0.1:35456 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 07:10:00.959275 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 07:10:00 dist-test-slave-14ww krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:00.960155 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 07:10:00 dist-test-slave-14ww krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:00.960994 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 07:10:00 dist-test-slave-14ww krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 07:10:00.967374 6442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:00.961491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35470 (local address 127.0.98.1:36203)
0924 07:10:00.961524 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:00.961526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:00.961578 (+ 52us) server_negotiation.cc:407] Connection header received
0924 07:10:00.961628 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:00.961628 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:00.961635 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:00.961682 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:00.962004 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:00.962444 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:00.963070 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:00.963248 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:00.963296 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:00.965272 (+ 1976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:00.965273 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:00.965273 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:00.965279 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:00.966424 (+ 1145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:00.966735 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:00.966736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:00.966736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:00.966784 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:00.966947 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:00.966947 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:00.966948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:00.967031 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:00.967096 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:00.967180 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 07:10:00.967233 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 07:10:00.968894 6442 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:00.968282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35478 (local address 127.0.98.1:36203)
0924 07:10:00.968340 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:00.968341 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:00.968414 (+ 73us) server_negotiation.cc:407] Connection header received
0924 07:10:00.968500 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:00.968501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:00.968510 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:00.968565 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:00.968852 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35478: BlockingRecv error: recv got EOF from 127.0.0.1:35478 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260924 07:10:00.969029 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790233772407728-392-0/krb5kdc/krb5cc)
Sep 24 07:10:00 dist-test-slave-14ww krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:00.969945 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 07:10:00 dist-test-slave-14ww krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:00.970784 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 07:10:00 dist-test-slave-14ww krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 07:10:00.977356 6442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:00.971238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35490 (local address 127.0.98.1:36203)
0924 07:10:00.971292 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:00.971293 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:00.971302 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:10:00.971367 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:00.971368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:00.971375 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:00.971419 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:00.971744 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:00.972201 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:00.972907 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:00.973101 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:00.973162 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:00.974977 (+ 1815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:00.974979 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:00.974979 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:00.974984 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:00.976349 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:00.976674 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:00.976675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:00.976675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:00.976721 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:00.976906 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:00.976906 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:00.976906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:00.977000 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:00.977064 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:00.977163 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 07:10:00.977214 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 07:10:00.977998 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 6038
I20260924 07:10:00.983500 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 6172
I20260924 07:10:00.987891 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 6306
I20260924 07:10:00.994136 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 5944
2026-09-24T07:10:00Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1870 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:10:01 dist-test-slave-14ww krb5kdc[6455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:10:01 dist-test-slave-14ww krb5kdc[6455](info): set up 2 sockets
Sep 24 07:10:01 dist-test-slave-14ww krb5kdc[6455](info): commencing operation
krb5kdc: starting...
W20260924 07:10:02.037075 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:10:02 dist-test-slave-14ww krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:10:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:10:02Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 07:10:02.152665 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35621
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35621
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:02.231189 6468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:02.231429 6468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:02.231463 6468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:02.232570 6468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:10:02.232643 6468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:02.232671 6468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:02.232689 6468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:10:02.232705 6468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:10:02.234146 6468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:10:02.234439 6468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:02.234656 6468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:02.240151 6476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:02.240178 6473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:02.240211 6468 server_base.cc:1034] running on GCE node
W20260924 07:10:02.240231 6474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:02.240696 6468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:02.241055 6468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:02.242221 6468 hybrid_clock.cc:648] HybridClock initialized: now 1790233802242210 us; error 32 us; skew 500 ppm
Sep 24 07:10:02 dist-test-slave-14ww krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233802, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:02.244692 6468 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 07:10:02.245143 6468 webserver.cc:460] Webserver started at http://127.0.98.62:43299/ using document root <none> and password file <none>
I20260924 07:10:02.245324 6468 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:02.245363 6468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:02.245451 6468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:02.246480 6468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "0556c444f6d740be8b98241bb6a3ebeb"
format_stamp: "Formatted at 2026-09-24 07:10:02 on dist-test-slave-14ww"
server_key: "7566540c4bdc3e63916855674448efff"
server_key_iv: "51889092d67506b2528cfbf54ce79fa8"
server_key_version: "encryptionkey@0"
I20260924 07:10:02.246843 6468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "0556c444f6d740be8b98241bb6a3ebeb"
format_stamp: "Formatted at 2026-09-24 07:10:02 on dist-test-slave-14ww"
server_key: "7566540c4bdc3e63916855674448efff"
server_key_iv: "51889092d67506b2528cfbf54ce79fa8"
server_key_version: "encryptionkey@0"
I20260924 07:10:02.248124 6468 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 07:10:02.248868 6483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:02.249023 6468 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:02.249125 6468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "0556c444f6d740be8b98241bb6a3ebeb"
format_stamp: "Formatted at 2026-09-24 07:10:02 on dist-test-slave-14ww"
server_key: "7566540c4bdc3e63916855674448efff"
server_key_iv: "51889092d67506b2528cfbf54ce79fa8"
server_key_version: "encryptionkey@0"
I20260924 07:10:02.249179 6468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:02.259745 6468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:02.261705 6468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:02.261807 6468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:02.265367 6468 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35621
I20260924 07:10:02.265442 6535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35621 every 8 connection(s)
I20260924 07:10:02.265743 6468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:10:02.266680 6536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:02.267069 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 6468
I20260924 07:10:02.267196 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:10:02.267395 392 external_mini_cluster.cc:1428] Setting key 5f4c7e2661f61449bb427f4d6e62c5d5
I20260924 07:10:02.269891 6536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb: Bootstrap starting.
I20260924 07:10:02.270471 6536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:02.270784 6536 log.cc:824] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb: Log is configured to *not* fsync() on all Append() calls
I20260924 07:10:02.271525 6536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb: No bootstrap required, opened a new log
I20260924 07:10:02.274011 6536 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } }
I20260924 07:10:02.274138 6536 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:02.274160 6536 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0556c444f6d740be8b98241bb6a3ebeb, State: Initialized, Role: FOLLOWER
I20260924 07:10:02.274286 6536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } }
I20260924 07:10:02.274363 6536 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:10:02.274385 6536 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:10:02.274449 6536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [term 0 FOLLOWER]: Advancing to term 1
Sep 24 07:10:02 dist-test-slave-14ww krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 07:10:02.275141 6536 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } }
I20260924 07:10:02.275262 6536 leader_election.cc:302] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0556c444f6d740be8b98241bb6a3ebeb; no voters:
I20260924 07:10:02.275453 6536 leader_election.cc:288] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:10:02.275586 6541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:10:02.275708 6536 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:10:02.275727 6541 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [term 1 LEADER]: Becoming Leader. State: Replica: 0556c444f6d740be8b98241bb6a3ebeb, State: Running, Role: LEADER
I20260924 07:10:02.275818 6541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } }
I20260924 07:10:02.277557 6543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0556c444f6d740be8b98241bb6a3ebeb. Latest consensus state: current_term: 1 leader_uuid: "0556c444f6d740be8b98241bb6a3ebeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } }
I20260924 07:10:02.277686 6543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:02.277559 6542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0556c444f6d740be8b98241bb6a3ebeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } }
I20260924 07:10:02.277781 6542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [sys.catalog]: This master's current role is: LEADER
W20260924 07:10:02.277917 6555 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:10:02.277973 6555 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:10:02.278105 6557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:10:02.278301 6557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:10:02.279166 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:02.267882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41410 (local address 127.0.98.62:35621)
0924 07:10:02.268050 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:02.268057 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:02.268086 (+ 29us) server_negotiation.cc:407] Connection header received
0924 07:10:02.268484 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:02.268490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:02.268532 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:02.268647 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:02.269627 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.270648 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.271647 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.272071 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.272159 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:02.275011 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:02.275014 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:02.275014 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:02.275024 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:02.277328 (+ 2304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:02.277757 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:02.277760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:02.277761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:02.277861 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:02.278103 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:02.278103 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:02.278104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:02.278436 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:02.278588 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:02.278826 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 07:10:02.278981 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":88,"threads_started":1}
I20260924 07:10:02.280836 6557 catalog_manager.cc:1347] Generated new cluster ID: c6d426aa15ca4806959a3a3058567fa2
I20260924 07:10:02.280896 6557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:10:02.291070 6557 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:10:02.291586 6557 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:10:02.295945 6557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb: Generated new TSK 0
I20260924 07:10:02.296116 6557 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 07:10:02.323915 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35621
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:02.407961 6562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:02.408155 6562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:02.408178 6562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:02.409327 6562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:02.409365 6562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:02.409391 6562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:10:02.410737 6562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:10:02.410916 6562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:02.411085 6562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:02.416046 6567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:02.416039 6568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:02.416052 6570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:02.416096 6562 server_base.cc:1034] running on GCE node
I20260924 07:10:02.416446 6562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:02.416714 6562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:02.417893 6562 hybrid_clock.cc:648] HybridClock initialized: now 1790233802417880 us; error 32 us; skew 500 ppm
Sep 24 07:10:02 dist-test-slave-14ww krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233802, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:02.420343 6562 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 07:10:02.420854 6562 webserver.cc:460] Webserver started at http://127.0.98.1:37221/ using document root <none> and password file <none>
I20260924 07:10:02.421092 6562 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:02.421159 6562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:02.421298 6562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:02.422297 6562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-0/data/instance:
uuid: "f6fe375836fc4922aecfa3904368afc1"
format_stamp: "Formatted at 2026-09-24 07:10:02 on dist-test-slave-14ww"
server_key: "0194ec9152666b252fcaa258c8996fc7"
server_key_iv: "82cd4be0465737bccff6268154ee047e"
server_key_version: "encryptionkey@0"
I20260924 07:10:02.422655 6562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f6fe375836fc4922aecfa3904368afc1"
format_stamp: "Formatted at 2026-09-24 07:10:02 on dist-test-slave-14ww"
server_key: "0194ec9152666b252fcaa258c8996fc7"
server_key_iv: "82cd4be0465737bccff6268154ee047e"
server_key_version: "encryptionkey@0"
I20260924 07:10:02.423938 6562 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 07:10:02.424628 6577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:02.424841 6562 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:10:02.424937 6562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-0/wal
uuid: "f6fe375836fc4922aecfa3904368afc1"
format_stamp: "Formatted at 2026-09-24 07:10:02 on dist-test-slave-14ww"
server_key: "0194ec9152666b252fcaa258c8996fc7"
server_key_iv: "82cd4be0465737bccff6268154ee047e"
server_key_version: "encryptionkey@0"
I20260924 07:10:02.425022 6562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:02.431067 6562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:02.432682 6562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:02.432852 6562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:02.433050 6562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:10:02.433329 6562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:10:02.433382 6562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:02.433421 6562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:10:02.433449 6562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:02.439352 6562 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45969
I20260924 07:10:02.439399 6690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45969 every 8 connection(s)
I20260924 07:10:02.439668 6562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 07:10:02 dist-test-slave-14ww krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233802, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 07:10:02.448447 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 6562
I20260924 07:10:02.448549 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:10:02.448638 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:02.440263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43841 (local address 127.0.98.62:35621)
0924 07:10:02.440320 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:02.440321 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:02.440524 (+ 203us) server_negotiation.cc:407] Connection header received
0924 07:10:02.441116 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:02.441118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:02.441129 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:02.441185 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:02.442143 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.442663 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.443639 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.443850 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.443937 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:02.446105 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:02.446106 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:02.446106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:02.446120 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:02.447369 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:02.447734 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:02.447735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:02.447735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:02.447782 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:02.447979 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:02.447980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:02.447980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:02.448087 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:02.448136 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:02.448432 (+ 296us) server_negotiation.cc:299] Negotiation successful
0924 07:10:02.448495 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 07:10:02.448777 392 external_mini_cluster.cc:1428] Setting key 2bbec6bb784c410f05e08872e2b345ed
I20260924 07:10:02.448853 6693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:02.440237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.98.1:43841)
0924 07:10:02.440411 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:02.440443 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:02.441012 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:02.441258 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:02.441261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:02.441868 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:02.442043 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:02.442047 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.442758 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.442759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:02.443545 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:02.443547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.443946 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.443947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:02.444065 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:02.444563 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:02.444571 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:02.446012 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:02.447439 (+ 1427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:02.447445 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:02.447446 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:02.447649 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:02.447838 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:02.447839 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:02.447839 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:02.447927 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:02.448141 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:02.448145 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:02.448292 (+ 147us) client_negotiation.cc:769] Sending connection context
0924 07:10:02.448415 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 07:10:02.448584 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":90,"threads_started":1}
I20260924 07:10:02.449323 6691 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35621
I20260924 07:10:02.449436 6691 heartbeater.cc:464] Registering TS with master...
I20260924 07:10:02.449615 6691 heartbeater.cc:511] Master 127.0.98.62:35621 requested a full tablet report, sending...
I20260924 07:10:02.450059 6500 ts_manager.cc:194] Registered new tserver with Master: f6fe375836fc4922aecfa3904368afc1 (127.0.98.1:45969)
I20260924 07:10:02.450654 6500 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43841
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 07:10:02.473410 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35621
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:02.553550 6696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:02.553727 6696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:02.553750 6696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:02.554824 6696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:02.554859 6696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:02.554885 6696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:10:02.556198 6696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:10:02.556357 6696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:02.556512 6696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:02.561261 6704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:02.561262 6701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:02.561362 6702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:02.561457 6696 server_base.cc:1034] running on GCE node
I20260924 07:10:02.561686 6696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:02.561923 6696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:02.563124 6696 hybrid_clock.cc:648] HybridClock initialized: now 1790233802563105 us; error 30 us; skew 500 ppm
Sep 24 07:10:02 dist-test-slave-14ww krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233802, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:02.565749 6696 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 07:10:02.566200 6696 webserver.cc:460] Webserver started at http://127.0.98.2:44221/ using document root <none> and password file <none>
I20260924 07:10:02.566437 6696 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:02.566504 6696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:02.566635 6696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:02.567605 6696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-1/data/instance:
uuid: "f74dc4545b8e4f7abc5ed2aa102c5cc5"
format_stamp: "Formatted at 2026-09-24 07:10:02 on dist-test-slave-14ww"
server_key: "874e3dcc22480df15875c30421ba078d"
server_key_iv: "21e986951d27ba2ba4180e9feef6c70a"
server_key_version: "encryptionkey@0"
I20260924 07:10:02.567953 6696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f74dc4545b8e4f7abc5ed2aa102c5cc5"
format_stamp: "Formatted at 2026-09-24 07:10:02 on dist-test-slave-14ww"
server_key: "874e3dcc22480df15875c30421ba078d"
server_key_iv: "21e986951d27ba2ba4180e9feef6c70a"
server_key_version: "encryptionkey@0"
I20260924 07:10:02.569329 6696 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:10:02.570075 6711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:02.570312 6696 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:10:02.570407 6696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-1/wal
uuid: "f74dc4545b8e4f7abc5ed2aa102c5cc5"
format_stamp: "Formatted at 2026-09-24 07:10:02 on dist-test-slave-14ww"
server_key: "874e3dcc22480df15875c30421ba078d"
server_key_iv: "21e986951d27ba2ba4180e9feef6c70a"
server_key_version: "encryptionkey@0"
I20260924 07:10:02.570493 6696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:02.590904 6696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:02.592979 6696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:02.593154 6696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:02.593389 6696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:10:02.593688 6696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:10:02.593741 6696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:02.593781 6696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:10:02.593809 6696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:02.599469 6696 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40657
I20260924 07:10:02.599503 6824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40657 every 8 connection(s)
I20260924 07:10:02.599825 6696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 07:10:02 dist-test-slave-14ww krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233802, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 07:10:02.607870 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 6696
I20260924 07:10:02.607973 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:10:02.608177 392 external_mini_cluster.cc:1428] Setting key ad6417e6086227db725fe92e0b902da7
I20260924 07:10:02.608577 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:02.600517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33599 (local address 127.0.98.62:35621)
0924 07:10:02.600563 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:02.600565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:02.600804 (+ 239us) server_negotiation.cc:407] Connection header received
0924 07:10:02.601285 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:02.601286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:02.601295 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:02.601343 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:02.602196 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.602689 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.603719 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.603899 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.603964 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:02.605975 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:02.605976 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:02.605976 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:02.605982 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:02.607194 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:02.607584 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:02.607585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:02.607585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:02.607631 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:02.607857 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:02.607858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:02.607859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:02.607999 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:02.608068 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:02.608391 (+ 323us) server_negotiation.cc:299] Negotiation successful
0924 07:10:02.608477 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 07:10:02.608769 6827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:02.600506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.98.2:33599)
0924 07:10:02.600658 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:02.600691 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:02.601205 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:02.601397 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:02.601400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:02.601935 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:02.602117 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:02.602121 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.602823 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.602824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:02.603639 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:02.603641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.603995 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.603996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:02.604100 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:02.604553 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:02.604561 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:02.605893 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:02.607268 (+ 1375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:02.607274 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:02.607275 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:02.607518 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:02.607702 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:02.607703 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:02.607703 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:02.607794 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:02.608098 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:02.608102 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:02.608230 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 07:10:02.608365 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 07:10:02.608537 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
I20260924 07:10:02.609206 6825 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35621
I20260924 07:10:02.609282 6825 heartbeater.cc:464] Registering TS with master...
I20260924 07:10:02.609469 6825 heartbeater.cc:511] Master 127.0.98.62:35621 requested a full tablet report, sending...
I20260924 07:10:02.609890 6500 ts_manager.cc:194] Registered new tserver with Master: f74dc4545b8e4f7abc5ed2aa102c5cc5 (127.0.98.2:40657)
I20260924 07:10:02.610260 6500 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33599
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 07:10:02.633037 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35621
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:02.712512 6830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:02.712749 6830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:02.712803 6830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:02.713948 6830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:02.714010 6830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:02.714054 6830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:10:02.715505 6830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:10:02.715756 6830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:02.715976 6830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:02.721148 6835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:02.721158 6838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:02.721158 6836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:02.721405 6830 server_base.cc:1034] running on GCE node
I20260924 07:10:02.721661 6830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:02.721932 6830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:02.723294 6830 hybrid_clock.cc:648] HybridClock initialized: now 1790233802723291 us; error 36 us; skew 500 ppm
Sep 24 07:10:02 dist-test-slave-14ww krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233802, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:02.725932 6830 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 07:10:02.726339 6830 webserver.cc:460] Webserver started at http://127.0.98.3:44461/ using document root <none> and password file <none>
I20260924 07:10:02.726573 6830 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:02.726619 6830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:02.726742 6830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:02.727849 6830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-2/data/instance:
uuid: "801287c2cafb4ad786ac93285bbe5080"
format_stamp: "Formatted at 2026-09-24 07:10:02 on dist-test-slave-14ww"
server_key: "856f32beb241970d1c88e9b777079f3b"
server_key_iv: "e5706b5dee4a6d83528450022867315d"
server_key_version: "encryptionkey@0"
I20260924 07:10:02.728264 6830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-2/wal/instance:
uuid: "801287c2cafb4ad786ac93285bbe5080"
format_stamp: "Formatted at 2026-09-24 07:10:02 on dist-test-slave-14ww"
server_key: "856f32beb241970d1c88e9b777079f3b"
server_key_iv: "e5706b5dee4a6d83528450022867315d"
server_key_version: "encryptionkey@0"
I20260924 07:10:02.729674 6830 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:10:02.730424 6845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:02.730638 6830 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 07:10:02.730724 6830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-2/wal
uuid: "801287c2cafb4ad786ac93285bbe5080"
format_stamp: "Formatted at 2026-09-24 07:10:02 on dist-test-slave-14ww"
server_key: "856f32beb241970d1c88e9b777079f3b"
server_key_iv: "e5706b5dee4a6d83528450022867315d"
server_key_version: "encryptionkey@0"
I20260924 07:10:02.730794 6830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:02.737915 6830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:02.739705 6830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:02.739851 6830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:02.740054 6830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:10:02.740338 6830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:10:02.740370 6830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:02.740389 6830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:10:02.740433 6830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:02.746348 6830 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43105
I20260924 07:10:02.746411 6958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43105 every 8 connection(s)
I20260924 07:10:02.746729 6830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:10:02.747308 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 6830
I20260924 07:10:02.747416 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:10:02.747584 392 external_mini_cluster.cc:1428] Setting key af451894986bbd2736a2c39d5d2db511
Sep 24 07:10:02 dist-test-slave-14ww krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233802, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 07:10:02.759473 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:02.747936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51991 (local address 127.0.98.62:35621)
0924 07:10:02.747976 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:02.747978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:02.748219 (+ 241us) server_negotiation.cc:407] Connection header received
0924 07:10:02.748775 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:02.748777 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:02.748789 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:02.748858 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:02.750037 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.750751 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.752298 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.752563 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.752639 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:02.755775 (+ 3136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:02.755776 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:02.755777 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:02.755784 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:02.757569 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:02.758074 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:02.758076 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:02.758076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:02.758144 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:02.758564 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:02.758565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:02.758566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:02.758781 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:02.758868 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:02.759296 (+ 428us) server_negotiation.cc:299] Negotiation successful
0924 07:10:02.759383 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 07:10:02.759610 6961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:02.747948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.98.3:51991)
0924 07:10:02.748100 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:02.748138 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:02.748657 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:02.748917 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:02.748920 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:02.749635 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:02.749905 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:02.749911 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.750837 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.750839 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:02.752204 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:02.752207 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.752702 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.752703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:02.752868 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:02.753603 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:02.753611 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:02.755671 (+ 2060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:02.757670 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:02.757676 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:02.757678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:02.757989 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:02.758322 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:02.758323 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:02.758323 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:02.758483 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:02.758925 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:02.758951 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:02.759126 (+ 175us) client_negotiation.cc:769] Sending connection context
0924 07:10:02.759238 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 07:10:02.759393 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
I20260924 07:10:02.762449 6959 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35621
I20260924 07:10:02.762573 6959 heartbeater.cc:464] Registering TS with master...
I20260924 07:10:02.762765 6959 heartbeater.cc:511] Master 127.0.98.62:35621 requested a full tablet report, sending...
I20260924 07:10:02.763319 6499 ts_manager.cc:194] Registered new tserver with Master: 801287c2cafb4ad786ac93285bbe5080 (127.0.98.3:43105)
I20260924 07:10:02.763913 6499 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51991
I20260924 07:10:02.771190 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260924 07:10:02.802067 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38773
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35621,127.0.98.61:38773
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:02.882881 6964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:02.883114 6964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:02.883137 6964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:02.884251 6964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:10:02.884289 6964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:02.884302 6964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:02.884315 6964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:10:02.884327 6964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:10:02.885776 6964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35621,127.0.98.61:38773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
W20260924 07:10:02.885880 6964 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:35621,127.0.98.61:38773'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 07:10:02.886039 6964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:02.886260 6964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:02.891666 6970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:02.891727 6964 server_base.cc:1034] running on GCE node
W20260924 07:10:02.891631 6969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:02.891631 6972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:02.892122 6964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:02.892419 6964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:02.893600 6964 hybrid_clock.cc:648] HybridClock initialized: now 1790233802893582 us; error 55 us; skew 500 ppm
Sep 24 07:10:02 dist-test-slave-14ww krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233802, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:02.896273 6964 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 07:10:02.896854 6964 webserver.cc:460] Webserver started at http://127.0.98.61:46521/ using document root <none> and password file <none>
I20260924 07:10:02.897091 6964 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:02.897156 6964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:02.897292 6964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:02.898412 6964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-1/data/instance:
uuid: "45abe618fb274f6db2ea422007f6c684"
format_stamp: "Formatted at 2026-09-24 07:10:02 on dist-test-slave-14ww"
server_key: "578ecc985ea4088e17240be5c7de2b48"
server_key_iv: "14da74e3f4e9eebf3eda00230b48b269"
server_key_version: "encryptionkey@0"
I20260924 07:10:02.898798 6964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-1/wal/instance:
uuid: "45abe618fb274f6db2ea422007f6c684"
format_stamp: "Formatted at 2026-09-24 07:10:02 on dist-test-slave-14ww"
server_key: "578ecc985ea4088e17240be5c7de2b48"
server_key_iv: "14da74e3f4e9eebf3eda00230b48b269"
server_key_version: "encryptionkey@0"
I20260924 07:10:02.900194 6964 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:10:02.901062 6979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:02.901247 6964 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:10:02.901347 6964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-1/wal
uuid: "45abe618fb274f6db2ea422007f6c684"
format_stamp: "Formatted at 2026-09-24 07:10:02 on dist-test-slave-14ww"
server_key: "578ecc985ea4088e17240be5c7de2b48"
server_key_iv: "14da74e3f4e9eebf3eda00230b48b269"
server_key_version: "encryptionkey@0"
I20260924 07:10:02.901436 6964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:02.910979 6964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:02.913074 6964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:02.913234 6964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:02.917815 6964 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:38773
I20260924 07:10:02.917866 7031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:38773 every 8 connection(s)
I20260924 07:10:02.918219 6964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-1/data/info.pb
I20260924 07:10:02.919379 7032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:02.920852 7032 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:10:02.926721 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 6964
Sep 24 07:10:02 dist-test-slave-14ww krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233802, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 07:10:02.931089 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:02.921552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41424 (local address 127.0.98.62:35621)
0924 07:10:02.921626 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:02.921629 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:02.921880 (+ 251us) server_negotiation.cc:407] Connection header received
0924 07:10:02.922376 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:02.922377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:02.922387 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:02.922441 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:02.923067 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.923563 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.924659 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.924931 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.924993 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:02.927746 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:02.927748 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:02.927749 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:02.927757 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:02.929521 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:02.930036 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:02.930037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:02.930037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:02.930090 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:02.930371 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:02.930371 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:02.930372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:02.930510 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:02.930565 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:02.930909 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 07:10:02.930998 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 07:10:02.931164 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:02.921534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.0.1:41424)
0924 07:10:02.921755 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:02.921794 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:02.922287 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:02.922504 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:02.922508 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:02.922759 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:02.922963 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:02.922967 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.923697 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.923698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:02.924557 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:02.924560 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.925011 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.925012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:02.925121 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:02.925716 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:02.925724 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:02.927585 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:02.929645 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:02.929650 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:02.929652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:02.929949 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:02.930163 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:02.930164 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:02.930164 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:02.930298 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:02.930579 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:02.930584 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:02.930748 (+ 164us) client_negotiation.cc:769] Sending connection context
0924 07:10:02.930846 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 07:10:02.930967 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":91,"threads_started":1}
I20260924 07:10:02.931809 7032 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:10:02 dist-test-slave-14ww krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 07:10:02 dist-test-slave-14ww krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233802, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 07:10:02.937738 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:02.927426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42502 (local address 127.0.98.61:38773)
0924 07:10:02.927582 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:02.927586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:02.927737 (+ 151us) server_negotiation.cc:407] Connection header received
0924 07:10:02.927764 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:02.927766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:02.927781 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:02.927872 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:02.928481 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.929616 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.930771 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.931217 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.931287 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:02.934259 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:02.934265 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:02.934266 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:02.934272 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:02.936125 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:02.936506 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:02.936509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:02.936509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:02.936587 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:02.936956 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:02.936956 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:02.936957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:02.937160 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:02.937337 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:02.937509 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 07:10:02.937593 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":72,"threads_started":1}
I20260924 07:10:02.940495 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:02.932114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38773 (local address 127.0.0.1:42518)
0924 07:10:02.932180 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:02.932192 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:02.932233 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:02.932482 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:02.932482 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:02.932653 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:02.932710 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:02.932712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.933806 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.933807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:02.934300 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:02.934302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.934820 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.934820 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:02.934896 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:02.935250 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:02.935254 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:02.936990 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:02.939594 (+ 2604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:02.939594 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:02.939595 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:02.939784 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:02.939966 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:02.939967 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:02.939967 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:02.940014 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:02.940244 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:02.940244 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:02.940287 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 07:10:02.940364 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 07:10:02.940436 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 07:10:02.940615 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:02.932208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42518 (local address 127.0.98.61:38773)
0924 07:10:02.932378 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:02.932379 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:02.932384 (+ 5us) server_negotiation.cc:407] Connection header received
0924 07:10:02.932398 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:02.932398 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:02.932405 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:02.932481 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:02.932830 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.933574 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.934379 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.934690 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.934773 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:02.937069 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:02.937070 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:02.937070 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:02.937076 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:02.939498 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:02.939854 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:02.939855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:02.939855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:02.939907 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:02.940052 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:02.940052 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:02.940052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:02.940148 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:02.940194 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:02.940387 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 07:10:02.940479 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":155,"thread_start_us":105,"threads_started":1}
I20260924 07:10:02.941973 7032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684: Bootstrap starting.
I20260924 07:10:02.942622 7032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:02.942900 7032 log.cc:824] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684: Log is configured to *not* fsync() on all Append() calls
I20260924 07:10:02.943650 7032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684: No bootstrap required, opened a new log
I20260924 07:10:02.945076 7032 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } }
I20260924 07:10:02.945220 7032 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:02.945268 7032 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45abe618fb274f6db2ea422007f6c684, State: Initialized, Role: FOLLOWER
I20260924 07:10:02.945382 7032 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } }
I20260924 07:10:02.945691 7032 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:10:02.945662 7039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } } }
I20260924 07:10:02.945760 7039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 07:10:02.947391 7050 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:10:02.947448 7050 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:10:02.949084 6964 master_runner.cc:428] Detected that this master 45abe618fb274f6db2ea422007f6c684 is joining an existing cluster
I20260924 07:10:02.949172 6964 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:38773
I20260924 07:10:02.959617 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:02.950074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38773 (local address 127.0.0.1:42532)
0924 07:10:02.953004 (+ 2930us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:02.953013 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:02.953059 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:02.953206 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:02.953207 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:02.953479 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:02.953547 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:02.953549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.954481 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.954482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:02.955328 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:02.955331 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.955767 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.955768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:02.955840 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:02.956347 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:02.956351 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:02.956897 (+ 546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:02.958564 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:02.958564 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:02.958565 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:02.958748 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:02.958948 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:02.958949 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:02.958949 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:02.959002 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:02.959258 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:02.959259 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:02.959295 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 07:10:02.959388 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 07:10:02.959491 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2917,"thread_start_us":68,"threads_started":1}
I20260924 07:10:02.959703 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:02.950207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42532 (local address 127.0.98.61:38773)
0924 07:10:02.950238 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:02.950239 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:02.953085 (+ 2846us) server_negotiation.cc:407] Connection header received
0924 07:10:02.953106 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:02.953107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:02.953118 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:02.953174 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:02.953664 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.954382 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.955423 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.955638 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.955716 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:02.956989 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:02.956990 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:02.956990 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:02.956996 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:02.958447 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:02.958820 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:02.958821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:02.958821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:02.958873 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:02.959072 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:02.959073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:02.959073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:02.959193 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:02.959266 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:02.959422 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 07:10:02.959590 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":15}
I20260924 07:10:02.960611 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:02.950106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41428 (local address 127.0.98.62:35621)
0924 07:10:02.950147 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:02.950149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:02.952981 (+ 2832us) server_negotiation.cc:407] Connection header received
0924 07:10:02.953008 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:02.953009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:02.953021 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:02.953120 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:02.953611 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.954395 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.955706 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.956002 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.956072 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:02.956986 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:02.956988 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:02.956989 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:02.956997 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:02.959334 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:02.959774 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:02.959775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:02.959776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:02.959833 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:02.960044 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:02.960045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:02.960046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:02.960197 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:02.960260 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:02.960452 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 07:10:02.960525 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 07:10:02.960634 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:02.950043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.0.1:41428)
0924 07:10:02.952839 (+ 2796us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:02.952851 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:02.952918 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:02.953193 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:02.953194 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:02.953454 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:02.953518 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:02.953519 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.954468 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.954469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:02.955616 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:02.955619 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.956071 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.956072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:02.956130 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:02.956480 (+ 350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:02.956485 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:02.956898 (+ 413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:02.959429 (+ 2531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:02.959430 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:02.959431 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:02.959710 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:02.959908 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:02.959909 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:02.959909 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:02.959972 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:02.960288 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:02.960289 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:02.960326 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 07:10:02.960425 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 07:10:02.960520 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2772,"mutex_wait_us":81,"thread_start_us":78,"threads_started":1}
Sep 24 07:10:02 dist-test-slave-14ww krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233802, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 07:10:02.970105 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:02.962744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38773 (local address 127.0.0.1:42544)
0924 07:10:02.962933 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:02.962965 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:02.963021 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:02.963155 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:02.963157 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:02.963386 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:02.963431 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:02.963432 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.964307 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.964308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:02.964945 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:02.964947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.965413 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.965414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:02.965480 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:02.965949 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:02.965954 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:02.967287 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:02.969106 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:02.969113 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:02.969115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:02.969312 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:02.969516 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:02.969516 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:02.969517 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:02.969569 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:02.969823 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:02.969828 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:02.969872 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 07:10:02.969937 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 07:10:02.970025 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":87,"threads_started":1}
I20260924 07:10:02.970144 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:02.962816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42544 (local address 127.0.98.61:38773)
0924 07:10:02.962868 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:02.962870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:02.963023 (+ 153us) server_negotiation.cc:407] Connection header received
0924 07:10:02.963076 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:02.963077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:02.963086 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:02.963153 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:02.963504 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.964229 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.965018 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.965309 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.965382 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:02.967370 (+ 1988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:02.967371 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:02.967371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:02.967378 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:02.969025 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:02.969384 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:02.969385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:02.969386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:02.969443 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:02.969629 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:02.969630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:02.969630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:02.969760 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:02.969827 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:02.969979 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 07:10:02.970052 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 07:10:02.970700 6499 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:38773: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "45abe618fb274f6db2ea422007f6c684"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 38773
}
attrs {
promote: true
}
}
dest_uuid: "0556c444f6d740be8b98241bb6a3ebeb"
cas_config_opid_index: -1
I20260924 07:10:02.971009 6499 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: true } }
I20260924 07:10:02.971482 7060 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 45abe618fb274f6db2ea422007f6c684 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: true } } }
I20260924 07:10:02.971774 7006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:02.972503 7060 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:38773
I20260924 07:10:02.972756 6964 master.cc:537] Master@127.0.98.61:38773 shutting down...
I20260924 07:10:02.972816 7006 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [term 1 FOLLOWER]: Refusing update from remote peer 0556c444f6d740be8b98241bb6a3ebeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 07:10:02.973106 7060 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:10:02.972793 7061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0556c444f6d740be8b98241bb6a3ebeb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: true } } }
I20260924 07:10:02.973258 7061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [sys.catalog]: This master's current role is: LEADER
W20260924 07:10:02.973743 6485 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260924 07:10:02.974540 6485 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb -> Peer 45abe618fb274f6db2ea422007f6c684 (127.0.98.61:38773): Couldn't send request to peer 45abe618fb274f6db2ea422007f6c684. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 07:10:02.975672 6964 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:10:02.975772 6964 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:10:02.975857 6964 tablet_replica.cc:330] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684: stopping tablet replica
I20260924 07:10:02.988914 6964 master.cc:552] Master@127.0.98.61:38773 shutdown complete.
I20260924 07:10:02.988996 6964 master_runner.cc:305] Clearing existing system tablet
I20260924 07:10:02.989143 6964 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:02.989192 6964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:02.989892 6964 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:10:02.990361 7065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:02.990482 6964 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:02.990538 6964 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:02.990569 6964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-1/wal
uuid: "45abe618fb274f6db2ea422007f6c684"
format_stamp: "Formatted at 2026-09-24 07:10:02 on dist-test-slave-14ww"
server_key: "578ecc985ea4088e17240be5c7de2b48"
server_key_iv: "14da74e3f4e9eebf3eda00230b48b269"
server_key_version: "encryptionkey@0"
I20260924 07:10:02.991182 6964 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:10:02.992691 6964 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 07:10:02.992822 6964 log.cc:1196] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 07:10:02.993026 6964 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684: Deleting consensus metadata
I20260924 07:10:02.993170 6964 master_runner.cc:315] Copying system tablet from 127.0.98.62:35621
I20260924 07:10:02.993793 6964 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:35621
I20260924 07:10:03.000705 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:02.994307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41440 (local address 127.0.98.62:35621)
0924 07:10:02.994360 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:02.994363 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:02.994461 (+ 98us) server_negotiation.cc:407] Connection header received
0924 07:10:02.994494 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:02.994495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:02.994551 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:02.994630 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:02.994977 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.995775 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.996492 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.996848 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.996914 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:02.998057 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:02.998058 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:02.998059 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:02.998067 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:02.999555 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:02.999942 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:02.999943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:02.999943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:02.999992 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.000206 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.000207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.000207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.000327 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:03.000392 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:03.000551 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 07:10:03.000640 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 07:10:03.000779 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:02.994203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.0.1:41440)
0924 07:10:02.994380 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:02.994390 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:02.994451 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:02.994673 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:02.994674 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:02.994839 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:02.994894 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:02.994896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.995857 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.995858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:02.996411 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:02.996412 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:02.996939 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:02.996940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:02.996996 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:02.997528 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:02.997534 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:02.997944 (+ 410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:02.999641 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:02.999642 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:02.999642 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:02.999861 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:03.000051 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:03.000052 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:03.000052 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:03.000116 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:03.000401 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:03.000402 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:03.000449 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 07:10:03.000533 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 07:10:03.000633 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":67,"threads_started":1}
I20260924 07:10:03.001518 6520 tablet_copy_service.cc:140] P 0556c444f6d740be8b98241bb6a3ebeb: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 45abe618fb274f6db2ea422007f6c684 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:41440)
I20260924 07:10:03.001623 6520 tablet_copy_service.cc:161] P 0556c444f6d740be8b98241bb6a3ebeb: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 45abe618fb274f6db2ea422007f6c684 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:41440: session id = 45abe618fb274f6db2ea422007f6c684-00000000000000000000000000000000
I20260924 07:10:03.002425 6520 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 07:10:03.003041 6964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 07:10:03.003978 6964 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684: tablet copy: Starting download of 0 data blocks...
I20260924 07:10:03.004149 6964 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684: tablet copy: Starting download of 1 WAL segments...
I20260924 07:10:03.005458 6964 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 07:10:03.006338 6520 tablet_copy_service.cc:342] P 0556c444f6d740be8b98241bb6a3ebeb: Request end of tablet copy session 45abe618fb274f6db2ea422007f6c684-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:41440
I20260924 07:10:03.006405 6520 tablet_copy_service.cc:433] P 0556c444f6d740be8b98241bb6a3ebeb: ending tablet copy session 45abe618fb274f6db2ea422007f6c684-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 45abe618fb274f6db2ea422007f6c684
I20260924 07:10:03.008697 6964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:03.010000 7072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:03.010144 7075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:03.010157 7073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:03.010371 6964 server_base.cc:1034] running on GCE node
I20260924 07:10:03.010473 6964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:03.010599 6964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:03.011734 6964 hybrid_clock.cc:648] HybridClock initialized: now 1790233803011723 us; error 24 us; skew 500 ppm
Sep 24 07:10:03 dist-test-slave-14ww krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233803, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:03.013223 6964 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 07:10:03.013504 6964 webserver.cc:460] Webserver started at http://127.0.98.61:45265/ using document root <none> and password file <none>
I20260924 07:10:03.013624 6964 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:03.013671 6964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:03.014387 6964 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:10:03.014947 7082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:03.015094 6964 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:10:03.015143 6964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-1/wal
uuid: "45abe618fb274f6db2ea422007f6c684"
format_stamp: "Formatted at 2026-09-24 07:10:02 on dist-test-slave-14ww"
server_key: "578ecc985ea4088e17240be5c7de2b48"
server_key_iv: "14da74e3f4e9eebf3eda00230b48b269"
server_key_version: "encryptionkey@0"
I20260924 07:10:03.015199 6964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:03.024883 6964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:03.026646 6964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:03.029704 6964 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:38773
I20260924 07:10:03.030112 6964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-1/data/info.pb
I20260924 07:10:03.030108 7134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:38773 every 8 connection(s)
I20260924 07:10:03.030807 7135 sys_catalog.cc:263] Verifying existing consensus state
I20260924 07:10:03.031204 7135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684: Bootstrap starting.
I20260924 07:10:03.033427 7135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:10:03.033690 7135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684: Bootstrap complete.
I20260924 07:10:03.033892 7135 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: true } }
I20260924 07:10:03.033989 7135 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 45abe618fb274f6db2ea422007f6c684, State: Initialized, Role: LEARNER
I20260924 07:10:03.034025 7135 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: true } }
I20260924 07:10:03.034189 7135 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:10:03.034181 7138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: true } } }
I20260924 07:10:03.034298 7138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [sys.catalog]: This master's current role is: LEARNER
I20260924 07:10:03.035610 7149 catalog_manager.cc:1259] Loaded cluster ID: c6d426aa15ca4806959a3a3058567fa2
I20260924 07:10:03.035655 7149 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684: loading cluster ID for follower catalog manager: success
I20260924 07:10:03.036612 7149 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684: acquiring CA information for follower catalog manager: success
I20260924 07:10:03.036911 7149 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 07:10:03.046161 7150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.039777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42556 (local address 127.0.98.61:38773)
0924 07:10:03.039887 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:03.039889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:03.039896 (+ 7us) server_negotiation.cc:407] Connection header received
0924 07:10:03.039924 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:03.039925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:03.039933 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:03.039988 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:03.040542 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.041263 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.041998 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.042197 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.042271 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.043500 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:03.043502 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:03.043502 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:03.043509 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:03.045003 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.045359 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.045360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.045360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.045419 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.045637 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.045638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.045639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.045742 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:03.045871 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:03.045955 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 07:10:03.046037 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":54,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260924 07:10:03.076000 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37159
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35621,127.0.98.61:38773,127.0.98.60:37159
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:03.168093 7153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:03.168291 7153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:03.168313 7153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:03.169544 7153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:10:03.169592 7153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:03.169607 7153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:03.169620 7153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:10:03.169631 7153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:10:03.171092 7153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35621,127.0.98.61:38773,127.0.98.60:37159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:10:03.171335 7153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:03.171546 7153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:03.176659 7159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:03.176721 7158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:03.176659 7161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:03.177048 7153 server_base.cc:1034] running on GCE node
I20260924 07:10:03.177292 7153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:03.177531 7153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:03.178808 7153 hybrid_clock.cc:648] HybridClock initialized: now 1790233803178778 us; error 33 us; skew 500 ppm
Sep 24 07:10:03 dist-test-slave-14ww krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233803, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:03.181821 7153 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 07:10:03.182268 7153 webserver.cc:460] Webserver started at http://127.0.98.60:36933/ using document root <none> and password file <none>
I20260924 07:10:03.182472 7153 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:03.182513 7153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:03.182607 7153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:03.183605 7153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-2/data/instance:
uuid: "f57386635f2c41e28f6b57747c1e62af"
format_stamp: "Formatted at 2026-09-24 07:10:03 on dist-test-slave-14ww"
server_key: "23df9825d13f3cbafe2d62790cd212bb"
server_key_iv: "fd74f7cc5267ba0d50ab87d08d04f41b"
server_key_version: "encryptionkey@0"
I20260924 07:10:03.183907 7153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-2/wal/instance:
uuid: "f57386635f2c41e28f6b57747c1e62af"
format_stamp: "Formatted at 2026-09-24 07:10:03 on dist-test-slave-14ww"
server_key: "23df9825d13f3cbafe2d62790cd212bb"
server_key_iv: "fd74f7cc5267ba0d50ab87d08d04f41b"
server_key_version: "encryptionkey@0"
I20260924 07:10:03.185421 7153 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:10:03.186575 7168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:03.186860 7153 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:10:03.186966 7153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-2/wal
uuid: "f57386635f2c41e28f6b57747c1e62af"
format_stamp: "Formatted at 2026-09-24 07:10:03 on dist-test-slave-14ww"
server_key: "23df9825d13f3cbafe2d62790cd212bb"
server_key_iv: "fd74f7cc5267ba0d50ab87d08d04f41b"
server_key_version: "encryptionkey@0"
I20260924 07:10:03.187060 7153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:03.201490 7153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:03.203802 7153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:03.203982 7153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:03.209514 7153 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37159
I20260924 07:10:03.209561 7220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37159 every 8 connection(s)
I20260924 07:10:03.209908 7153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-2/data/info.pb
I20260924 07:10:03.210863 7221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:03.211274 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 7153
I20260924 07:10:03.213264 7221 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:10:03 dist-test-slave-14ww krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 07:10:03 dist-test-slave-14ww krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233803, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 07:10:03.222365 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.213980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41446 (local address 127.0.98.62:35621)
0924 07:10:03.214100 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:03.214103 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:03.214205 (+ 102us) server_negotiation.cc:407] Connection header received
0924 07:10:03.214745 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:03.214746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:03.214758 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:03.214807 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:03.215293 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.215798 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.216826 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.217154 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.217229 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.219477 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:03.219478 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:03.219479 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:03.219488 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:03.221144 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.221522 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.221524 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.221524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.221590 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.221813 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.221814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.221815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.221929 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:03.222004 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:03.222195 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 07:10:03.222267 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260924 07:10:03.222582 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.213846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.0.1:41446)
0924 07:10:03.214083 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:03.214129 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:03.214639 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:03.214832 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:03.214835 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:03.215147 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:03.215211 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.215213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.215905 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.215906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.216719 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.216722 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.217273 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.217274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.217331 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.217791 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:03.217798 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:03.219372 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:03.221242 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:03.221248 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:03.221250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:03.221434 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:03.221655 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:03.221656 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:03.221657 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:03.221744 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:03.221994 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:03.221999 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:03.222072 (+ 73us) client_negotiation.cc:769] Sending connection context
0924 07:10:03.222175 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 07:10:03.222380 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":65,"thread_start_us":98,"threads_started":1}
I20260924 07:10:03.222587 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.211860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57432 (local address 127.0.98.60:37159)
0924 07:10:03.212022 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:03.212030 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:03.212058 (+ 28us) server_negotiation.cc:407] Connection header received
0924 07:10:03.212472 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:03.212478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:03.212521 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:03.212703 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:03.213333 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.214469 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.215262 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.215670 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.215785 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.218586 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:03.218592 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:03.218592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:03.218599 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:03.221023 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.221429 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.221432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.221433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.221517 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.221723 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.221723 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.221724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.221971 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:03.222124 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:03.222202 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 07:10:03.222314 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":71,"threads_started":1}
I20260924 07:10:03.223079 7221 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:10:03 dist-test-slave-14ww krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233803, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 07:10:03.229386 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.223316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38773 (local address 127.0.0.1:42564)
0924 07:10:03.223363 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:03.223372 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:03.223405 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:03.223529 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:03.223530 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:03.223630 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:03.223670 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.223671 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.224261 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.224262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.224818 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.224820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.225195 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.225196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.225246 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.225604 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:03.225608 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:03.226673 (+ 1065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:03.228465 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:03.228466 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:03.228466 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:03.228627 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:03.228860 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:03.228861 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:03.228861 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:03.228937 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:03.229182 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:03.229182 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:03.229216 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 07:10:03.229267 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 07:10:03.229331 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 07:10:03.229401 7150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.223377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42564 (local address 127.0.98.61:38773)
0924 07:10:03.223415 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:03.223417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:03.223426 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:10:03.223473 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:03.223473 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:03.223483 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:03.223529 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:03.223728 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.224179 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.224913 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.225125 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.225180 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.226776 (+ 1596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:03.226778 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:03.226778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:03.226785 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:03.228392 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.228703 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.228704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.228704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.228797 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.229018 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.229018 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.229019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.229124 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:03.229176 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:03.229284 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 07:10:03.229342 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 07:10:03.229761 7221 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37159 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:10:03 dist-test-slave-14ww krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233803, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 07:10:03.235908 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.229988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37159 (local address 127.0.0.1:57440)
0924 07:10:03.230034 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:03.230044 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:03.230088 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:03.230207 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:03.230208 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:03.230326 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:03.230348 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.230349 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.230981 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.230981 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.231433 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.231434 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.231804 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.231804 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.231851 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.232214 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:03.232218 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:03.233359 (+ 1141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:03.234878 (+ 1519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:03.234878 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:03.234879 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:03.235031 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:03.235317 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:03.235318 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:03.235318 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:03.235381 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:03.235581 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:03.235582 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:03.235636 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 07:10:03.235703 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 07:10:03.235772 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 07:10:03.235942 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.230040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57440 (local address 127.0.98.60:37159)
0924 07:10:03.230067 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:03.230069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:03.230076 (+ 7us) server_negotiation.cc:407] Connection header received
0924 07:10:03.230143 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:03.230144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:03.230152 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:03.230203 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:03.230427 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.230892 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.231514 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.231722 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.231803 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.233494 (+ 1691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:03.233495 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:03.233495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:03.233500 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:03.234811 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.235129 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.235130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.235130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.235234 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.235436 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.235437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.235437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.235526 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:03.235595 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:03.235720 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 07:10:03.235845 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":13}
I20260924 07:10:03.237246 7221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af: Bootstrap starting.
I20260924 07:10:03.238029 7221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:03.238355 7221 log.cc:824] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af: Log is configured to *not* fsync() on all Append() calls
I20260924 07:10:03.239050 7221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af: No bootstrap required, opened a new log
I20260924 07:10:03.240423 7221 raft_consensus.cc:348] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } } peers { permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37159 } }
I20260924 07:10:03.240545 7221 raft_consensus.cc:374] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:03.240566 7221 raft_consensus.cc:729] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f57386635f2c41e28f6b57747c1e62af, State: Initialized, Role: FOLLOWER
I20260924 07:10:03.240713 7221 consensus_queue.cc:260] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } } peers { permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37159 } }
I20260924 07:10:03.240999 7226 sys_catalog.cc:455] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } } peers { permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37159 } } }
I20260924 07:10:03.241065 7221 sys_catalog.cc:564] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:10:03.241081 7226 sys_catalog.cc:458] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [sys.catalog]: This master's current role is: FOLLOWER
W20260924 07:10:03.242784 7237 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:10:03.242838 7237 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:10:03.245016 7153 master_runner.cc:428] Detected that this master f57386635f2c41e28f6b57747c1e62af is joining an existing cluster
I20260924 07:10:03.245056 7153 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:37159
I20260924 07:10:03.253810 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.246007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37159 (local address 127.0.0.1:57448)
0924 07:10:03.246230 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:03.246240 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:03.246307 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:03.246483 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:03.246484 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:03.246758 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:03.246837 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.246839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.247787 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.247788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.249076 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.249079 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.249549 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.249549 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.249620 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.250140 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:03.250144 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:03.250440 (+ 296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:03.252222 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:03.252223 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:03.252223 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:03.252931 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:03.253154 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:03.253155 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:03.253155 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:03.253210 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:03.253460 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:03.253461 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:03.253496 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 07:10:03.253594 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 07:10:03.253688 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":92,"threads_started":1}
I20260924 07:10:03.253813 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.246202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57448 (local address 127.0.98.60:37159)
0924 07:10:03.246231 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:03.246233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:03.246293 (+ 60us) server_negotiation.cc:407] Connection header received
0924 07:10:03.246388 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:03.246389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:03.246397 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:03.246426 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:03.246915 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.247630 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.249172 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.249429 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.249464 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.250589 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:03.250590 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:03.250590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:03.250597 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:03.252132 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.253005 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.253006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.253006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.253086 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.253288 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.253289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.253289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.253386 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:03.253456 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:03.253618 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 07:10:03.253734 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":18}
I20260924 07:10:03.273375 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.246011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41454 (local address 127.0.98.62:35621)
0924 07:10:03.246054 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:03.246056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:03.246470 (+ 414us) server_negotiation.cc:407] Connection header received
0924 07:10:03.246561 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:03.246563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:03.246574 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:03.246605 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:03.247020 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.247726 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.249281 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.249577 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.249616 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.270117 (+ 20501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:03.270119 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:03.270119 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:03.270127 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:03.272142 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.272540 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.272541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.272541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.272601 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.272814 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.272815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.272816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.272943 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:03.273022 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:03.273189 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 07:10:03.273290 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 07:10:03.273378 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.246111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.0.1:41454)
0924 07:10:03.246420 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:03.246428 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:03.246491 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:03.246666 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:03.246666 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:03.246892 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:03.246942 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.246943 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.247812 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.247812 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.249196 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.249197 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.249688 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.249689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.249754 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.269537 (+ 19783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:03.269542 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:03.269999 (+ 457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:03.272227 (+ 2228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:03.272228 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:03.272228 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:03.272474 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:03.272679 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:03.272679 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:03.272679 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:03.272759 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:03.273013 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:03.273014 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:03.273060 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 07:10:03.273142 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 07:10:03.273263 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":53,"threads_started":1}
I20260924 07:10:03.273867 7150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.246197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42570 (local address 127.0.98.61:38773)
0924 07:10:03.246236 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:03.246238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:03.246322 (+ 84us) server_negotiation.cc:407] Connection header received
0924 07:10:03.246381 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:03.246382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:03.246392 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:03.246463 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:03.246971 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.247696 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.249233 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.249522 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.249562 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.270195 (+ 20633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:03.270197 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:03.270197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:03.270206 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:03.271973 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.272434 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.272435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.272435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.272514 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.272783 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.272784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.272785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.272936 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:03.273035 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:03.273695 (+ 660us) server_negotiation.cc:299] Negotiation successful
0924 07:10:03.273781 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 07:10:03.274132 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.246023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38773 (local address 127.0.0.1:42570)
0924 07:10:03.246275 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:03.246284 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:03.246332 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:03.246522 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:03.246523 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:03.246812 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:03.246877 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.246879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.247792 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.247806 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.249125 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.249127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.249635 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.249636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.249701 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.269468 (+ 19767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:03.269474 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:03.270034 (+ 560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:03.272072 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:03.272073 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:03.272073 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:03.272365 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:03.272588 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:03.272588 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:03.272589 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:03.272688 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:03.273471 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:03.273472 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:03.273525 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 07:10:03.273940 (+ 415us) client_negotiation.cc:240] Negotiation successful
0924 07:10:03.274043 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":67,"threads_started":1}
I20260924 07:10:03.274837 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 6562
I20260924 07:10:03.281798 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45969
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35621
--webserver_port=37221
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:35621,127.0.98.61:38773
--tserver_master_addrs=127.0.98.62:35621,127.0.98.61:38773,127.0.98.60:37159 with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:10:03 dist-test-slave-14ww krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233802, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 07:10:03.290930 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.280345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57460 (local address 127.0.98.60:37159)
0924 07:10:03.280437 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:03.280439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:03.280451 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:10:03.280487 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:03.280488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:03.280499 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:03.280576 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:03.281001 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.281702 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.282584 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.282793 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.282882 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.285523 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:03.285525 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:03.285525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:03.285532 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:03.288710 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.289441 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.289442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.289443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.289509 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.290226 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.290227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.290227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.290402 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:03.290519 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:03.290744 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 07:10:03.290828 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 07:10:03.291365 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.276085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37159 (local address 127.0.0.1:57460)
0924 07:10:03.276174 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:03.276184 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:03.276221 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:03.280601 (+ 4380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:03.280603 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:03.280850 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:03.280908 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.280910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.281800 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.281801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.282480 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.282483 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.282883 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.282884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.282962 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.283483 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:03.283489 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:03.285365 (+ 1876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:03.289078 (+ 3713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:03.289079 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:03.289080 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:03.289345 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:03.289933 (+ 588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:03.289934 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:03.289934 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:03.290022 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:03.290536 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:03.290538 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:03.290612 (+ 74us) client_negotiation.cc:769] Sending connection context
0924 07:10:03.290708 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 07:10:03.290842 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 07:10:03.291958 6499 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:37159: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f57386635f2c41e28f6b57747c1e62af"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 37159
}
attrs {
promote: true
}
}
dest_uuid: "0556c444f6d740be8b98241bb6a3ebeb"
cas_config_opid_index: 5
I20260924 07:10:03.292254 6499 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: true } } peers { permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37159 } attrs { promote: true } }
I20260924 07:10:03.292666 7060 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f57386635f2c41e28f6b57747c1e62af (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: true } } peers { permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37159 } attrs { promote: true } } }
I20260924 07:10:03.293795 7060 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:37159
I20260924 07:10:03.293874 7060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0556c444f6d740be8b98241bb6a3ebeb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: true } } peers { permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37159 } attrs { promote: true } } }
I20260924 07:10:03.293972 7060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:03.294412 7153 master.cc:537] Master@127.0.98.60:37159 shutting down...
I20260924 07:10:03.294562 7195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:03.295687 7195 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [term 1 FOLLOWER]: Refusing update from remote peer 0556c444f6d740be8b98241bb6a3ebeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 07:10:03.295979 7061 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37159 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 07:10:03.296581 6486 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb -> Peer f57386635f2c41e28f6b57747c1e62af (127.0.98.60:37159): Couldn't send request to peer f57386635f2c41e28f6b57747c1e62af. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 07:10:03.298653 7153 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:10:03.298735 7153 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:10:03.298777 7153 tablet_replica.cc:330] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af: stopping tablet replica
I20260924 07:10:03.298852 7150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.294259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42576 (local address 127.0.98.61:38773)
0924 07:10:03.294304 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:03.294306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:03.294890 (+ 584us) server_negotiation.cc:407] Connection header received
0924 07:10:03.294990 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:03.294991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:03.295013 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:03.295052 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:10:03.295238 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.296011 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.297788 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.298379 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.298424 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.298505 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:03.298744 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 07:10:03.298782 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 07:10:03.299167 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.294192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38773 (local address 127.0.0.1:42576)
0924 07:10:03.294832 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:03.294844 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:03.294919 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:03.295126 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:03.295126 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:03.295128 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:10:03.295168 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.295170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.296107 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.296108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.297530 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.297533 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.298565 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.298565 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.298643 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.298666 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 07:10:03.299058 (+ 392us) client_negotiation.cc:240] Negotiation successful
0924 07:10:03.299103 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622}
I20260924 07:10:03.299405 7109 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [term 1 LEARNER]: Refusing update from remote peer 0556c444f6d740be8b98241bb6a3ebeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 07:10:03.299598 7061 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 07:10:03.300128 7109 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f57386635f2c41e28f6b57747c1e62af (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: true } } peers { permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37159 } attrs { promote: true } } }
I20260924 07:10:03.301192 7138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0556c444f6d740be8b98241bb6a3ebeb. Latest consensus state: current_term: 1 leader_uuid: "0556c444f6d740be8b98241bb6a3ebeb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: true } } peers { permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37159 } attrs { promote: true } } }
I20260924 07:10:03.301267 7138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [sys.catalog]: This master's current role is: LEARNER
I20260924 07:10:03.301296 7138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0556c444f6d740be8b98241bb6a3ebeb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: true } } peers { permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37159 } attrs { promote: true } } }
I20260924 07:10:03.301331 7138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [sys.catalog]: This master's current role is: LEARNER
I20260924 07:10:03.301978 7061 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb: attempting to promote NON_VOTER 45abe618fb274f6db2ea422007f6c684 to VOTER
I20260924 07:10:03.302175 7061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37159 } attrs { promote: true } }
I20260924 07:10:03.302628 7109 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [term 1 LEARNER]: Refusing update from remote peer 0556c444f6d740be8b98241bb6a3ebeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 07:10:03.302759 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.302657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37159 (local address 127.0.0.1:57470)
0924 07:10:03.302698 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:03.302732 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 07:10:03.302927 7060 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 07:10:03.303165 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.303050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37159 (local address 127.0.0.1:57480)
0924 07:10:03.303123 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:03.303144 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37159: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260924 07:10:03.303222 6486 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb -> Peer f57386635f2c41e28f6b57747c1e62af (127.0.98.60:37159): Couldn't send request to peer f57386635f2c41e28f6b57747c1e62af. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37159: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 07:10:03.303598 7060 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 45abe618fb274f6db2ea422007f6c684 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37159 } attrs { promote: true } } }
I20260924 07:10:03.303754 7109 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 45abe618fb274f6db2ea422007f6c684 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37159 } attrs { promote: true } } }
I20260924 07:10:03.304406 7060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0556c444f6d740be8b98241bb6a3ebeb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37159 } attrs { promote: true } } }
I20260924 07:10:03.304473 7060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:03.304522 7060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0556c444f6d740be8b98241bb6a3ebeb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37159 } attrs { promote: true } } }
I20260924 07:10:03.304556 7060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:03.304874 7248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0556c444f6d740be8b98241bb6a3ebeb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37159 } attrs { promote: true } } }
I20260924 07:10:03.304998 7248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:10:03.312165 7153 master.cc:552] Master@127.0.98.60:37159 shutdown complete.
I20260924 07:10:03.312217 7153 master_runner.cc:305] Clearing existing system tablet
I20260924 07:10:03.312342 7153 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:03.312387 7153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:03.313158 7153 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:10:03.313702 7250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:03.313846 7153 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:03.313908 7153 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:03.313931 7153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-2/wal
uuid: "f57386635f2c41e28f6b57747c1e62af"
format_stamp: "Formatted at 2026-09-24 07:10:03 on dist-test-slave-14ww"
server_key: "23df9825d13f3cbafe2d62790cd212bb"
server_key_iv: "fd74f7cc5267ba0d50ab87d08d04f41b"
server_key_version: "encryptionkey@0"
I20260924 07:10:03.314594 7153 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:10:03.316159 7153 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 07:10:03.316226 7153 log.cc:1196] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 07:10:03.316406 7153 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af: Deleting consensus metadata
I20260924 07:10:03.316520 7153 master_runner.cc:315] Copying system tablet from 127.0.98.62:35621
I20260924 07:10:03.317113 7153 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:35621
I20260924 07:10:03.324178 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.317431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41460 (local address 127.0.98.62:35621)
0924 07:10:03.317471 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:03.317473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:03.317543 (+ 70us) server_negotiation.cc:407] Connection header received
0924 07:10:03.317609 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:03.317610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:03.317620 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:03.317679 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:03.318026 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.318801 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.319790 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.320076 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.320148 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.321295 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:03.321297 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:03.321297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:03.321305 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:03.322968 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.323359 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.323360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.323360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.323420 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.323645 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.323646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.323647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.323778 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:03.323849 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:03.324021 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 07:10:03.324093 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 07:10:03.324180 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.317375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.0.1:41460)
0924 07:10:03.317490 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:03.317500 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:03.317556 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:03.317692 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:03.317693 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:03.317893 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:03.317946 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.317949 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.318899 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.318899 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.319694 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.319697 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.320165 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.320166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.320232 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.320805 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:03.320811 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:03.321201 (+ 390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:03.323058 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:03.323059 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:03.323059 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:03.323276 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:03.323499 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:03.323500 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:03.323500 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:03.323567 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:03.323857 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:03.323858 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:03.323904 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 07:10:03.323984 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 07:10:03.324084 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":61,"threads_started":1}
I20260924 07:10:03.324797 6520 tablet_copy_service.cc:140] P 0556c444f6d740be8b98241bb6a3ebeb: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f57386635f2c41e28f6b57747c1e62af ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:41460)
I20260924 07:10:03.324858 6520 tablet_copy_service.cc:161] P 0556c444f6d740be8b98241bb6a3ebeb: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f57386635f2c41e28f6b57747c1e62af at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:41460: session id = f57386635f2c41e28f6b57747c1e62af-00000000000000000000000000000000
I20260924 07:10:03.325398 6520 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 07:10:03.325815 7153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 07:10:03.326941 7153 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af: tablet copy: Starting download of 0 data blocks...
I20260924 07:10:03.327077 7153 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af: tablet copy: Starting download of 1 WAL segments...
I20260924 07:10:03.327806 7153 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 07:10:03.328577 6520 tablet_copy_service.cc:342] P 0556c444f6d740be8b98241bb6a3ebeb: Request end of tablet copy session f57386635f2c41e28f6b57747c1e62af-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:41460
I20260924 07:10:03.328641 6520 tablet_copy_service.cc:433] P 0556c444f6d740be8b98241bb6a3ebeb: ending tablet copy session f57386635f2c41e28f6b57747c1e62af-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f57386635f2c41e28f6b57747c1e62af
I20260924 07:10:03.330996 7153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:03.332219 7257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:03.333784 7260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:03.333863 7153 server_base.cc:1034] running on GCE node
W20260924 07:10:03.333668 7258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:03.334159 7153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:03.334302 7153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:03.335412 7153 hybrid_clock.cc:648] HybridClock initialized: now 1790233803335388 us; error 39 us; skew 500 ppm
Sep 24 07:10:03 dist-test-slave-14ww krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233803, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:03.336855 7153 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 07:10:03.337106 7153 webserver.cc:460] Webserver started at http://127.0.98.60:42363/ using document root <none> and password file <none>
I20260924 07:10:03.337224 7153 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:03.337262 7153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:03.337965 7153 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:10:03.338774 7267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:03.338943 7153 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:10:03.338999 7153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-2/wal
uuid: "f57386635f2c41e28f6b57747c1e62af"
format_stamp: "Formatted at 2026-09-24 07:10:03 on dist-test-slave-14ww"
server_key: "23df9825d13f3cbafe2d62790cd212bb"
server_key_iv: "fd74f7cc5267ba0d50ab87d08d04f41b"
server_key_version: "encryptionkey@0"
I20260924 07:10:03.339053 7153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:03.362615 7153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:03.364450 7153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:03.367528 7153 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37159
I20260924 07:10:03.367617 7319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37159 every 8 connection(s)
I20260924 07:10:03.367859 7153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/master-2/data/info.pb
I20260924 07:10:03.369453 7320 sys_catalog.cc:263] Verifying existing consensus state
I20260924 07:10:03.369941 7320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af: Bootstrap starting.
I20260924 07:10:03.373013 7320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:10:03.373219 7320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af: Bootstrap complete.
I20260924 07:10:03.373414 7320 raft_consensus.cc:348] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37159 } attrs { promote: true } }
I20260924 07:10:03.373503 7320 raft_consensus.cc:729] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f57386635f2c41e28f6b57747c1e62af, State: Initialized, Role: LEARNER
I20260924 07:10:03.373539 7320 consensus_queue.cc:260] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37159 } attrs { promote: true } }
I20260924 07:10:03.373726 7320 sys_catalog.cc:564] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:10:03.373813 7323 sys_catalog.cc:455] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37159 } attrs { promote: true } } }
I20260924 07:10:03.373999 7323 sys_catalog.cc:458] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [sys.catalog]: This master's current role is: LEARNER
I20260924 07:10:03.375315 7334 catalog_manager.cc:1259] Loaded cluster ID: c6d426aa15ca4806959a3a3058567fa2
I20260924 07:10:03.375370 7334 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af: loading cluster ID for follower catalog manager: success
I20260924 07:10:03.376649 7334 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af: acquiring CA information for follower catalog manager: success
I20260924 07:10:03.376948 7334 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 07:10:03.383987 7246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:03.384212 7246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:03.384262 7246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:03.385462 7246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:03.385525 7246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:03.385576 7246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:10:03.387073 7246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45969
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37221
--tserver_master_addrs=127.0.98.62:35621,127.0.98.61:38773,127.0.98.60:37159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:10:03.387326 7246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:03.387543 7246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:03.393163 7339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:03.393144 7340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:03.393163 7342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:03.393778 7246 server_base.cc:1034] running on GCE node
I20260924 07:10:03.393982 7246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:03.394222 7246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:03.395390 7246 hybrid_clock.cc:648] HybridClock initialized: now 1790233803395376 us; error 26 us; skew 500 ppm
Sep 24 07:10:03 dist-test-slave-14ww krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233803, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:03.398192 7246 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 07:10:03.398721 7246 webserver.cc:460] Webserver started at http://127.0.98.1:37221/ using document root <none> and password file <none>
I20260924 07:10:03.399026 7246 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:03.399145 7246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:03.400519 7246 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 07:10:03.401199 7349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:03.401377 7246 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 07:10:03.401477 7246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-0/wal
uuid: "f6fe375836fc4922aecfa3904368afc1"
format_stamp: "Formatted at 2026-09-24 07:10:02 on dist-test-slave-14ww"
server_key: "0194ec9152666b252fcaa258c8996fc7"
server_key_iv: "82cd4be0465737bccff6268154ee047e"
server_key_version: "encryptionkey@0"
I20260924 07:10:03.401832 7246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:03.427132 7246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:03.429507 7246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:03.429674 7246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:03.429910 7246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:10:03.430228 7246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:10:03.430284 7246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:03.430318 7246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:10:03.430346 7246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:03.436905 7246 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45969
I20260924 07:10:03.436944 7462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45969 every 8 connection(s)
I20260924 07:10:03.437387 7246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 07:10:03 dist-test-slave-14ww krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233803, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 07:10:03 dist-test-slave-14ww krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233803, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 07:10:03 dist-test-slave-14ww krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233803, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 07:10:03.447546 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 7246
I20260924 07:10:03.447707 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 6696
I20260924 07:10:03.448478 7150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.438346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41369 (local address 127.0.98.61:38773)
0924 07:10:03.438392 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:03.438394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:03.438638 (+ 244us) server_negotiation.cc:407] Connection header received
0924 07:10:03.439187 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:03.439189 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:03.439203 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:03.439257 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:03.440155 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.440927 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.442272 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.442484 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.442565 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.445129 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:03.445130 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:03.445131 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:03.445138 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:03.446953 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.447344 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.447345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.447346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.447402 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.447718 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.447719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.447720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.447859 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:03.447904 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:03.448293 (+ 389us) server_negotiation.cc:299] Negotiation successful
0924 07:10:03.448376 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 07:10:03.448632 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.438120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44381 (local address 127.0.98.62:35621)
0924 07:10:03.438282 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:03.438285 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:03.438792 (+ 507us) server_negotiation.cc:407] Connection header received
0924 07:10:03.439318 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:03.439319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:03.439331 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:03.439379 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:03.440156 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.440927 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.442223 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.442537 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.442596 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.445640 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:03.445642 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:03.445642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:03.445650 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:03.447273 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.447812 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.447813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.447814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.447863 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.448061 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.448062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.448063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.448185 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:03.448228 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:03.448452 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 07:10:03.448526 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260924 07:10:03.449054 7470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.438068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.98.1:44381)
0924 07:10:03.438705 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:03.438714 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:03.439064 (+ 350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:03.439413 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:03.439414 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:03.439848 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:03.440039 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.440051 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.441018 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.441019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.442141 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.442142 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.442641 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.442642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.442707 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.443458 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:03.443461 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:03.445487 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:03.447497 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:03.447497 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:03.447498 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:03.447731 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:03.447935 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:03.447936 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:03.447936 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:03.447991 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:03.448283 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:03.448284 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:03.448333 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 07:10:03.448706 (+ 373us) client_negotiation.cc:240] Negotiation successful
0924 07:10:03.448854 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":420,"thread_start_us":61,"threads_started":1}
I20260924 07:10:03.449059 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.438068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38773 (local address 127.0.98.1:41369)
0924 07:10:03.438537 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:03.438587 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:03.439059 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:03.439356 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:03.439361 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:03.439849 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:03.440039 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.440045 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.441018 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.441019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.442135 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.442137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.442575 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.442576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.442701 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.443531 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:03.443535 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:03.445038 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:03.447021 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:03.447028 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:03.447030 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:03.447267 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:03.447489 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:03.447490 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:03.447491 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:03.447630 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:03.447952 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:03.447958 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:03.448123 (+ 165us) client_negotiation.cc:769] Sending connection context
0924 07:10:03.448612 (+ 489us) client_negotiation.cc:240] Negotiation successful
0924 07:10:03.448809 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":92,"thread_start_us":83,"threads_started":1}
I20260924 07:10:03.449636 7465 heartbeater.cc:347] Connected to a master server at 127.0.98.61:38773
I20260924 07:10:03.449713 7465 heartbeater.cc:464] Registering TS with master...
I20260924 07:10:03.449842 7465 heartbeater.cc:511] Master 127.0.98.61:38773 requested a full tablet report, sending...
I20260924 07:10:03.450326 7099 ts_manager.cc:194] Registered new tserver with Master: f6fe375836fc4922aecfa3904368afc1 (127.0.98.1:45969)
I20260924 07:10:03.450346 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.438069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37159 (local address 127.0.98.1:42393)
0924 07:10:03.438364 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:03.438408 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:03.439059 (+ 651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:03.439377 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:03.439378 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:03.439849 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:03.440038 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.440043 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.440981 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.440982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.441977 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.441979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.442619 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.442620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.442715 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.443439 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:03.443449 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:03.445910 (+ 2461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:03.448287 (+ 2377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:03.448288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:03.448289 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:03.448506 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:03.449552 (+ 1046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:03.449553 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:03.449553 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:03.449620 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:03.450016 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:03.450017 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:03.450067 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 07:10:03.450161 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 07:10:03.450247 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":175,"thread_start_us":92,"threads_started":1}
I20260924 07:10:03.450680 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.438174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42393 (local address 127.0.98.60:37159)
0924 07:10:03.438282 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:03.438285 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:03.438583 (+ 298us) server_negotiation.cc:407] Connection header received
0924 07:10:03.439215 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:03.439216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:03.439236 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:03.439322 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:03.440138 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.440881 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.442089 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.442489 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.442558 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.445996 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:03.445998 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:03.445998 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:03.446005 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:03.448189 (+ 2184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.449036 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.449037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.449038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.449475 (+ 437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.449729 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.449763 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.449764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.449955 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:03.450024 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:03.450187 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 07:10:03.450290 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":64,"threads_started":1}
I20260924 07:10:03.451083 7463 heartbeater.cc:347] Connected to a master server at 127.0.98.60:37159
I20260924 07:10:03.451135 7463 heartbeater.cc:464] Registering TS with master...
I20260924 07:10:03.451206 7463 heartbeater.cc:511] Master 127.0.98.60:37159 requested a full tablet report, sending...
I20260924 07:10:03.451550 7284 ts_manager.cc:194] Registered new tserver with Master: f6fe375836fc4922aecfa3904368afc1 (127.0.98.1:45969)
I20260924 07:10:03.453713 7464 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35621
I20260924 07:10:03.453771 7464 heartbeater.cc:464] Registering TS with master...
I20260924 07:10:03.453837 7464 heartbeater.cc:511] Master 127.0.98.62:35621 requested a full tablet report, sending...
I20260924 07:10:03.454108 6499 ts_manager.cc:194] Re-registered known tserver with Master: f6fe375836fc4922aecfa3904368afc1 (127.0.98.1:45969)
I20260924 07:10:03.454487 6499 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44381
I20260924 07:10:03.454936 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40657
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35621
--webserver_port=44221
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:35621,127.0.98.61:38773
--tserver_master_addrs=127.0.98.62:35621,127.0.98.61:38773,127.0.98.60:37159 with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:03.535563 7471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:03.535781 7471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:03.535816 7471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:03.536981 7471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:03.537050 7471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:03.537097 7471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:10:03.538555 7471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40657
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44221
--tserver_master_addrs=127.0.98.62:35621,127.0.98.61:38773,127.0.98.60:37159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:10:03.538825 7471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:03.539114 7471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:03.544269 7479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:03.544236 7476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:03.544230 7477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:03.544499 7471 server_base.cc:1034] running on GCE node
I20260924 07:10:03.544694 7471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:03.544935 7471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:03.546096 7471 hybrid_clock.cc:648] HybridClock initialized: now 1790233803546079 us; error 30 us; skew 500 ppm
Sep 24 07:10:03 dist-test-slave-14ww krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233803, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:03.548450 7471 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 07:10:03.548945 7471 webserver.cc:460] Webserver started at http://127.0.98.2:44221/ using document root <none> and password file <none>
I20260924 07:10:03.549153 7471 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:03.549230 7471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:03.550458 7471 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:10:03.551005 7486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:03.551203 7471 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:10:03.551294 7471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-1/wal
uuid: "f74dc4545b8e4f7abc5ed2aa102c5cc5"
format_stamp: "Formatted at 2026-09-24 07:10:02 on dist-test-slave-14ww"
server_key: "874e3dcc22480df15875c30421ba078d"
server_key_iv: "21e986951d27ba2ba4180e9feef6c70a"
server_key_version: "encryptionkey@0"
I20260924 07:10:03.551601 7471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:03.566394 7471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:03.568277 7471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:03.568434 7471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:03.568650 7471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:10:03.569002 7471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:10:03.569057 7471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:03.569092 7471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:10:03.569119 7471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:03.575519 7471 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40657
I20260924 07:10:03.575649 7599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40657 every 8 connection(s)
I20260924 07:10:03.576052 7471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:10:03.579299 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 7471
I20260924 07:10:03.579453 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 6830
Sep 24 07:10:03 dist-test-slave-14ww krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233803, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 07:10:03 dist-test-slave-14ww krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233803, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 07:10:03 dist-test-slave-14ww krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233803, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 07:10:03.587239 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43105
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35621
--webserver_port=44461
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:35621,127.0.98.61:38773
--tserver_master_addrs=127.0.98.62:35621,127.0.98.61:38773,127.0.98.60:37159 with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:10:03.589764 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.577276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48805 (local address 127.0.98.60:37159)
0924 07:10:03.577318 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:03.577320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:03.577328 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:10:03.577995 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:03.578069 (+ 74us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:03.578092 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:03.578160 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:03.579073 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.579850 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.581781 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.582092 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.582129 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.585112 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:03.585114 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:03.585114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:03.585122 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:03.586750 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.588268 (+ 1518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.588270 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.588270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.588334 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.588621 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.588622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.588622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.588804 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:03.588869 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:03.589548 (+ 679us) server_negotiation.cc:299] Negotiation successful
0924 07:10:03.589655 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 07:10:03.589913 7150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.577305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34101 (local address 127.0.98.61:38773)
0924 07:10:03.577350 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:03.577352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:03.577546 (+ 194us) server_negotiation.cc:407] Connection header received
0924 07:10:03.578344 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:03.578345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:03.578358 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:03.578412 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:03.579074 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.579855 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.581782 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.582078 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.582119 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.586620 (+ 4501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:03.586622 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:03.586622 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:03.586631 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:03.588623 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.589007 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.589008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.589008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.589064 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.589275 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.589276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.589276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.589479 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:03.589537 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:03.589738 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 07:10:03.589815 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 07:10:03.590579 7605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.576982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37159 (local address 127.0.98.2:48805)
0924 07:10:03.577263 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:03.577275 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:03.577865 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:03.578250 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:03.578253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:03.578831 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:03.578965 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.578970 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.579981 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.579983 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.581674 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.581677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.582208 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.582209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.582266 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.583052 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:03.583056 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:03.585008 (+ 1952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:03.587484 (+ 2476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:03.587490 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:03.587493 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:03.588174 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:03.588411 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:03.588411 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:03.588412 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:03.588544 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:03.589094 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:03.589100 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:03.589285 (+ 185us) client_negotiation.cc:769] Sending connection context
0924 07:10:03.589906 (+ 621us) client_negotiation.cc:240] Negotiation successful
0924 07:10:03.590385 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":407,"thread_start_us":220,"threads_started":1}
I20260924 07:10:03.590603 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.576982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38773 (local address 127.0.98.2:34101)
0924 07:10:03.577468 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:03.577478 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:03.577875 (+ 397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:03.578438 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:03.578439 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:03.578733 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:03.578965 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.578970 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.580042 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.580042 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.581676 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.581677 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.582198 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.582199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.582266 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.582949 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:03.582957 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:03.586523 (+ 3566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:03.588705 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:03.588706 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:03.588706 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:03.588922 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:03.589135 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:03.589136 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:03.589136 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:03.589204 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:03.589572 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:03.589573 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:03.589613 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 07:10:03.589985 (+ 372us) client_negotiation.cc:240] Negotiation successful
0924 07:10:03.590305 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":548,"thread_start_us":81,"threads_started":1}
I20260924 07:10:03.590802 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.576994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57435 (local address 127.0.98.62:35621)
0924 07:10:03.577043 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:03.577045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:03.577294 (+ 249us) server_negotiation.cc:407] Connection header received
0924 07:10:03.578218 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:03.578219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:03.578229 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:03.578289 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:03.579060 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.579856 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.581578 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.581912 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.581960 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.587233 (+ 5273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:03.587235 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:03.587236 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:03.587244 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:03.589154 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.589595 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.589597 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.589597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.589887 (+ 290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.590158 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.590159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.590159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.590303 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:03.590353 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:03.590584 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 07:10:03.590669 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 07:10:03.591460 7604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.576982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.98.2:57435)
0924 07:10:03.577159 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:03.577210 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:03.577875 (+ 665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:03.578326 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:03.578327 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:03.578734 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:03.578963 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.578968 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.579985 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.579986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.581478 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.581481 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.582051 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.582052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.582211 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.583168 (+ 957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:03.583173 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:03.587123 (+ 3950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:03.589263 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:03.589264 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:03.589265 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:03.589502 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:03.589995 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:03.589996 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:03.589996 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:03.590066 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:03.590404 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:03.590405 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:03.590462 (+ 57us) client_negotiation.cc:769] Sending connection context
0924 07:10:03.590894 (+ 432us) client_negotiation.cc:240] Negotiation successful
0924 07:10:03.591172 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":106,"thread_start_us":92,"threads_started":1}
I20260924 07:10:03.592597 7602 heartbeater.cc:347] Connected to a master server at 127.0.98.61:38773
I20260924 07:10:03.592811 7602 heartbeater.cc:464] Registering TS with master...
I20260924 07:10:03.592856 7601 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35621
I20260924 07:10:03.592893 7601 heartbeater.cc:464] Registering TS with master...
I20260924 07:10:03.593166 7602 heartbeater.cc:511] Master 127.0.98.61:38773 requested a full tablet report, sending...
I20260924 07:10:03.593196 7600 heartbeater.cc:347] Connected to a master server at 127.0.98.60:37159
I20260924 07:10:03.593323 7600 heartbeater.cc:464] Registering TS with master...
I20260924 07:10:03.593408 7600 heartbeater.cc:511] Master 127.0.98.60:37159 requested a full tablet report, sending...
I20260924 07:10:03.593165 7601 heartbeater.cc:511] Master 127.0.98.62:35621 requested a full tablet report, sending...
I20260924 07:10:03.593848 7099 ts_manager.cc:194] Registered new tserver with Master: f74dc4545b8e4f7abc5ed2aa102c5cc5 (127.0.98.2:40657)
I20260924 07:10:03.593848 7284 ts_manager.cc:194] Registered new tserver with Master: f74dc4545b8e4f7abc5ed2aa102c5cc5 (127.0.98.2:40657)
I20260924 07:10:03.593786 6499 ts_manager.cc:194] Re-registered known tserver with Master: f74dc4545b8e4f7abc5ed2aa102c5cc5 (127.0.98.2:40657)
I20260924 07:10:03.594453 6499 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57435
W20260924 07:10:03.677045 7607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:03.677251 7607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:03.677289 7607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:03.678413 7607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:03.678455 7607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:03.678495 7607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:10:03.679852 7607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43105
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44461
--tserver_master_addrs=127.0.98.62:35621,127.0.98.61:38773,127.0.98.60:37159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:10:03.680022 7607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:03.680186 7607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:03.685256 7613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:03.685264 7615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:03.685271 7612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:03.685322 7607 server_base.cc:1034] running on GCE node
I20260924 07:10:03.685631 7607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:03.685861 7607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:03.687024 7607 hybrid_clock.cc:648] HybridClock initialized: now 1790233803687000 us; error 36 us; skew 500 ppm
Sep 24 07:10:03 dist-test-slave-14ww krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233803, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:03.689544 7607 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 07:10:03.689963 7607 webserver.cc:460] Webserver started at http://127.0.98.3:44461/ using document root <none> and password file <none>
I20260924 07:10:03.690210 7607 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:03.690291 7607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:03.691534 7607 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:10:03.692044 7622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:03.692221 7607 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:10:03.692313 7607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-2/wal
uuid: "801287c2cafb4ad786ac93285bbe5080"
format_stamp: "Formatted at 2026-09-24 07:10:02 on dist-test-slave-14ww"
server_key: "856f32beb241970d1c88e9b777079f3b"
server_key_iv: "e5706b5dee4a6d83528450022867315d"
server_key_version: "encryptionkey@0"
I20260924 07:10:03.692610 7607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:03.716500 7607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:03.718348 7607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:03.718504 7607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:03.718721 7607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:10:03.719007 7607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:10:03.719058 7607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:03.719098 7607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:10:03.719126 7607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:03.724723 7607 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43105
I20260924 07:10:03.724802 7735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43105 every 8 connection(s)
I20260924 07:10:03.725131 7607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:10:03.727706 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 7607
Sep 24 07:10:03 dist-test-slave-14ww krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233803, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 07:10:03 dist-test-slave-14ww krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233803, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 07:10:03.737325 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.725815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56853 (local address 127.0.98.60:37159)
0924 07:10:03.725862 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:03.725864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:03.726473 (+ 609us) server_negotiation.cc:407] Connection header received
0924 07:10:03.726773 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:03.726775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:03.726787 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:03.726818 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:03.727539 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.728252 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.729884 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.730182 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.730237 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.733528 (+ 3291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:03.733528 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:03.733529 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:03.733536 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:03.735819 (+ 2283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.736245 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.736246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.736246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.736303 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.736556 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.736557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.736557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.736682 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:03.736773 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:03.737119 (+ 346us) server_negotiation.cc:299] Negotiation successful
0924 07:10:03.737210 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 07:10:03.738749 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.725830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37159 (local address 127.0.98.3:56853)
0924 07:10:03.726396 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:03.726404 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:03.726675 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:03.727279 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:03.727280 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:03.727424 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:03.727479 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.727480 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.728491 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.728492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.729791 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.729793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.730291 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.730292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.730425 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.731132 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:03.731137 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:03.733441 (+ 2304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:03.735900 (+ 2459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:03.735905 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:03.735907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:03.736167 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:03.736367 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:03.736368 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:03.736368 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:03.736491 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:03.736797 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:03.736803 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:03.736959 (+ 156us) client_negotiation.cc:769] Sending connection context
0924 07:10:03.737078 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 07:10:03.738443 (+ 1365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":349,"thread_start_us":57,"threads_started":1}
I20260924 07:10:03.739203 7736 heartbeater.cc:347] Connected to a master server at 127.0.98.60:37159
I20260924 07:10:03.739298 7736 heartbeater.cc:464] Registering TS with master...
I20260924 07:10:03.739419 7150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.726124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36109 (local address 127.0.98.61:38773)
0924 07:10:03.726176 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:03.726178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:03.726338 (+ 160us) server_negotiation.cc:407] Connection header received
0924 07:10:03.726768 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:03.726769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:03.726781 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:03.726815 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:03.727484 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.728254 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.730097 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.730384 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.730439 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.734098 (+ 3659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:03.734099 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:03.734100 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:03.734107 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:03.736670 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.737932 (+ 1262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.737933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.737933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.737987 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.738182 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.738183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.738183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.738305 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:03.738363 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:03.739242 (+ 879us) server_negotiation.cc:299] Negotiation successful
0924 07:10:03.739314 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 07:10:03.739475 7736 heartbeater.cc:511] Master 127.0.98.60:37159 requested a full tablet report, sending...
I20260924 07:10:03.739643 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.728806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41468 (local address 127.0.98.62:35621)
0924 07:10:03.729115 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:03.729117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:03.729124 (+ 7us) server_negotiation.cc:407] Connection header received
0924 07:10:03.729143 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:03.729144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:03.729154 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:03.729216 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:03.729871 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.730638 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.731677 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.731976 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.732309 (+ 333us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.733496 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:03.733498 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:03.733498 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:03.733505 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:03.737954 (+ 4449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.738414 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.738415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.738416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.738490 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.738698 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.738698 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.738699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.738829 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:03.739282 (+ 453us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:03.739324 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 07:10:03.739549 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":81,"threads_started":1}
I20260924 07:10:03.738749 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.725830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38773 (local address 127.0.98.3:36109)
0924 07:10:03.726258 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:03.726266 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:03.726672 (+ 406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:03.726898 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:03.726900 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:03.727257 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:03.727378 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.727384 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.728372 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.728372 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.730022 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.730023 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.730487 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.730488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.730551 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.731021 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:03.731031 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:03.734011 (+ 2980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:03.736776 (+ 2765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:03.736777 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:03.736778 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:03.737845 (+ 1067us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:03.738057 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:03.738058 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:03.738058 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:03.738118 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:03.738368 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:03.738369 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:03.738415 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 07:10:03.738471 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 07:10:03.738590 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":756,"thread_start_us":74,"threads_started":1}
I20260924 07:10:03.739892 7284 ts_manager.cc:194] Registered new tserver with Master: 801287c2cafb4ad786ac93285bbe5080 (127.0.98.3:43105)
I20260924 07:10:03.740039 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.729025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42592 (local address 127.0.98.61:38773)
0924 07:10:03.729913 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:03.729915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:03.730826 (+ 911us) server_negotiation.cc:407] Connection header received
0924 07:10:03.730846 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:03.730847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:03.730858 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:03.730908 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:03.732402 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.733216 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.734160 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.734508 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.734565 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.736119 (+ 1554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:03.736120 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:03.736120 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:03.736126 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:03.738179 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.738637 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.738637 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.738638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.738690 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.738921 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.738922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.738923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.739416 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:03.739539 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:03.739848 (+ 309us) server_negotiation.cc:299] Negotiation successful
0924 07:10:03.739921 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":872,"thread_start_us":65,"threads_started":1}
I20260924 07:10:03.740263 7738 heartbeater.cc:347] Connected to a master server at 127.0.98.61:38773
I20260924 07:10:03.740309 7738 heartbeater.cc:464] Registering TS with master...
I20260924 07:10:03.740363 7738 heartbeater.cc:511] Master 127.0.98.61:38773 requested a full tablet report, sending...
I20260924 07:10:03.740572 7099 ts_manager.cc:194] Registered new tserver with Master: 801287c2cafb4ad786ac93285bbe5080 (127.0.98.3:43105)
I20260924 07:10:03.741333 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.729158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57486 (local address 127.0.98.60:37159)
0924 07:10:03.729958 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:03.729960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:03.729966 (+ 6us) server_negotiation.cc:407] Connection header received
0924 07:10:03.729983 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:03.729984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:03.729994 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:03.730051 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:03.730947 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.731669 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.733088 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.733364 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.733429 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.735239 (+ 1810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:03.735240 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:03.735241 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:03.735248 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:03.738202 (+ 2954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.739043 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.739044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.739044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.739098 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.740826 (+ 1728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.740827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.740827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.740940 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:03.741042 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:03.741148 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 07:10:03.741223 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":784,"thread_start_us":54,"threads_started":1}
Sep 24 07:10:03 dist-test-slave-14ww krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233803, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 07:10:03.741266 6499 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41468:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:10:03.743472 6499 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:10:03.744689 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.726121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35787 (local address 127.0.98.62:35621)
0924 07:10:03.726167 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:03.726170 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:03.726180 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:10:03.726758 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:03.726760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:03.726773 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:03.726817 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:03.727466 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.728246 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.736327 (+ 8081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.736616 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.736654 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.742240 (+ 5586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:03.742241 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:03.742241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:03.742247 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:03.743530 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.743880 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.743881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.743881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.743933 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.744167 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.744167 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.744168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.744308 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:03.744352 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:03.744573 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 07:10:03.744636 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 07:10:03.744946 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.725831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.98.3:35787)
0924 07:10:03.726008 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:03.726053 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:03.726653 (+ 600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:03.726894 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:03.726904 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:03.727202 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:03.727376 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.727380 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.728371 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.728372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.736238 (+ 7866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.736239 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.736746 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.736747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.736817 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.737303 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:03.737309 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:03.742164 (+ 4855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:03.743633 (+ 1469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:03.743634 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:03.743634 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:03.743806 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:03.744009 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:03.744009 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:03.744010 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:03.744066 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:03.744402 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:03.744403 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:03.744440 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 07:10:03.744512 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 07:10:03.744597 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":724,"thread_start_us":89,"threads_started":1}
I20260924 07:10:03.745309 7737 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35621
I20260924 07:10:03.745365 7737 heartbeater.cc:464] Registering TS with master...
I20260924 07:10:03.745435 7737 heartbeater.cc:511] Master 127.0.98.62:35621 requested a full tablet report, sending...
I20260924 07:10:03.745632 6500 ts_manager.cc:194] Re-registered known tserver with Master: 801287c2cafb4ad786ac93285bbe5080 (127.0.98.3:43105)
I20260924 07:10:03.746075 6500 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35787
I20260924 07:10:03.751215 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.747615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43105 (local address 127.0.0.1:60978)
0924 07:10:03.747658 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:03.747667 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:03.747702 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:03.748084 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:03.748085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:03.748086 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:10:03.748114 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.748115 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.749368 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.749369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.750245 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.750248 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.751024 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.751025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.751082 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.751086 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:10:03.751114 (+ 28us) client_negotiation.cc:240] Negotiation successful
0924 07:10:03.751152 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 07:10:03.751585 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.747780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60978 (local address 127.0.98.3:43105)
0924 07:10:03.747923 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:03.747928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:03.747937 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:10:03.747971 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:03.747973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:03.747987 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:03.748091 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:10:03.748198 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.749282 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.750318 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.750916 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.751329 (+ 413us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.751412 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:03.751436 (+ 24us) server_negotiation.cc:299] Negotiation successful
0924 07:10:03.751469 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
I20260924 07:10:03.751638 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.747714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40657 (local address 127.0.0.1:34334)
0924 07:10:03.747932 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:03.747942 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:03.747979 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:03.748366 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:03.748367 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:03.748369 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:10:03.748401 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.748403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.749317 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.749318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.750464 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.750466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.751400 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.751401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.751462 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.751466 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:10:03.751520 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 07:10:03.751554 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":61,"thread_start_us":87,"threads_started":1}
I20260924 07:10:03.751995 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.747713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45969 (local address 127.0.0.1:59224)
0924 07:10:03.747843 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:03.747852 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:03.747897 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:03.748406 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:03.748407 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:03.748409 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:10:03.748450 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.748452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.749831 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.749832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.750774 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.750776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.751767 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.751768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.751848 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.751852 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:10:03.751882 (+ 30us) client_negotiation.cc:240] Negotiation successful
0924 07:10:03.751923 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
I20260924 07:10:03.752312 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.747856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59224 (local address 127.0.98.1:45969)
0924 07:10:03.748236 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:03.748240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:03.748250 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:10:03.748286 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:03.748287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:03.748302 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:03.748535 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:10:03.748594 (+ 59us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.749745 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.750854 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.751672 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.752104 (+ 432us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.752204 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:03.752226 (+ 22us) server_negotiation.cc:299] Negotiation successful
0924 07:10:03.752257 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":90,"threads_started":1}
I20260924 07:10:03.752272 7670 tablet_service.cc:1467] Processing CreateTablet for tablet 918f5c329e8c43a08c0d72b9e8463c29 (DEFAULT_TABLE table=test-table [id=e3925939eeef46eb81810d730dd29b84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:10:03.752672 7670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 918f5c329e8c43a08c0d72b9e8463c29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:03.752957 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.748033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34334 (local address 127.0.98.2:40657)
0924 07:10:03.748216 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:03.748220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:03.748230 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:10:03.748271 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:03.748272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:03.748285 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:03.748369 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:10:03.748463 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.749224 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.750556 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.751271 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.751333 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.751830 (+ 497us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:03.752852 (+ 1022us) server_negotiation.cc:299] Negotiation successful
0924 07:10:03.752899 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":87,"threads_started":1}
I20260924 07:10:03.752945 7397 tablet_service.cc:1467] Processing CreateTablet for tablet 918f5c329e8c43a08c0d72b9e8463c29 (DEFAULT_TABLE table=test-table [id=e3925939eeef46eb81810d730dd29b84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:10:03.753273 7397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 918f5c329e8c43a08c0d72b9e8463c29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:03.756700 7762 tablet_bootstrap.cc:492] T 918f5c329e8c43a08c0d72b9e8463c29 P 801287c2cafb4ad786ac93285bbe5080: Bootstrap starting.
I20260924 07:10:03.757421 7762 tablet_bootstrap.cc:654] T 918f5c329e8c43a08c0d72b9e8463c29 P 801287c2cafb4ad786ac93285bbe5080: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:03.757316 7763 tablet_bootstrap.cc:492] T 918f5c329e8c43a08c0d72b9e8463c29 P f6fe375836fc4922aecfa3904368afc1: Bootstrap starting.
I20260924 07:10:03.757311 7534 tablet_service.cc:1467] Processing CreateTablet for tablet 918f5c329e8c43a08c0d72b9e8463c29 (DEFAULT_TABLE table=test-table [id=e3925939eeef46eb81810d730dd29b84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:10:03.757624 7534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 918f5c329e8c43a08c0d72b9e8463c29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:03.757745 7762 log.cc:824] T 918f5c329e8c43a08c0d72b9e8463c29 P 801287c2cafb4ad786ac93285bbe5080: Log is configured to *not* fsync() on all Append() calls
I20260924 07:10:03.758137 7763 tablet_bootstrap.cc:654] T 918f5c329e8c43a08c0d72b9e8463c29 P f6fe375836fc4922aecfa3904368afc1: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:03.758459 7763 log.cc:824] T 918f5c329e8c43a08c0d72b9e8463c29 P f6fe375836fc4922aecfa3904368afc1: Log is configured to *not* fsync() on all Append() calls
I20260924 07:10:03.758617 7762 tablet_bootstrap.cc:492] T 918f5c329e8c43a08c0d72b9e8463c29 P 801287c2cafb4ad786ac93285bbe5080: No bootstrap required, opened a new log
I20260924 07:10:03.758692 7762 ts_tablet_manager.cc:1397] T 918f5c329e8c43a08c0d72b9e8463c29 P 801287c2cafb4ad786ac93285bbe5080: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:10:03.759248 7763 tablet_bootstrap.cc:492] T 918f5c329e8c43a08c0d72b9e8463c29 P f6fe375836fc4922aecfa3904368afc1: No bootstrap required, opened a new log
I20260924 07:10:03.759315 7763 ts_tablet_manager.cc:1397] T 918f5c329e8c43a08c0d72b9e8463c29 P f6fe375836fc4922aecfa3904368afc1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:10:03.760458 7762 raft_consensus.cc:348] T 918f5c329e8c43a08c0d72b9e8463c29 P 801287c2cafb4ad786ac93285bbe5080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801287c2cafb4ad786ac93285bbe5080" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43105 } } peers { permanent_uuid: "f6fe375836fc4922aecfa3904368afc1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45969 } } peers { permanent_uuid: "f74dc4545b8e4f7abc5ed2aa102c5cc5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40657 } }
I20260924 07:10:03.760571 7762 raft_consensus.cc:374] T 918f5c329e8c43a08c0d72b9e8463c29 P 801287c2cafb4ad786ac93285bbe5080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:03.760591 7762 raft_consensus.cc:729] T 918f5c329e8c43a08c0d72b9e8463c29 P 801287c2cafb4ad786ac93285bbe5080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 801287c2cafb4ad786ac93285bbe5080, State: Initialized, Role: FOLLOWER
I20260924 07:10:03.760706 7762 consensus_queue.cc:260] T 918f5c329e8c43a08c0d72b9e8463c29 P 801287c2cafb4ad786ac93285bbe5080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801287c2cafb4ad786ac93285bbe5080" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43105 } } peers { permanent_uuid: "f6fe375836fc4922aecfa3904368afc1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45969 } } peers { permanent_uuid: "f74dc4545b8e4f7abc5ed2aa102c5cc5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40657 } }
I20260924 07:10:03.761000 7762 ts_tablet_manager.cc:1428] T 918f5c329e8c43a08c0d72b9e8463c29 P 801287c2cafb4ad786ac93285bbe5080: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 07:10:03.761296 7737 heartbeater.cc:503] Master 127.0.98.62:35621 was elected leader, sending a full tablet report...
I20260924 07:10:03.761958 7767 tablet_bootstrap.cc:492] T 918f5c329e8c43a08c0d72b9e8463c29 P f74dc4545b8e4f7abc5ed2aa102c5cc5: Bootstrap starting.
I20260924 07:10:03.761893 7763 raft_consensus.cc:348] T 918f5c329e8c43a08c0d72b9e8463c29 P f6fe375836fc4922aecfa3904368afc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801287c2cafb4ad786ac93285bbe5080" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43105 } } peers { permanent_uuid: "f6fe375836fc4922aecfa3904368afc1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45969 } } peers { permanent_uuid: "f74dc4545b8e4f7abc5ed2aa102c5cc5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40657 } }
I20260924 07:10:03.762053 7763 raft_consensus.cc:374] T 918f5c329e8c43a08c0d72b9e8463c29 P f6fe375836fc4922aecfa3904368afc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:03.762108 7763 raft_consensus.cc:729] T 918f5c329e8c43a08c0d72b9e8463c29 P f6fe375836fc4922aecfa3904368afc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6fe375836fc4922aecfa3904368afc1, State: Initialized, Role: FOLLOWER
I20260924 07:10:03.762233 7763 consensus_queue.cc:260] T 918f5c329e8c43a08c0d72b9e8463c29 P f6fe375836fc4922aecfa3904368afc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801287c2cafb4ad786ac93285bbe5080" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43105 } } peers { permanent_uuid: "f6fe375836fc4922aecfa3904368afc1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45969 } } peers { permanent_uuid: "f74dc4545b8e4f7abc5ed2aa102c5cc5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40657 } }
I20260924 07:10:03.762482 7763 ts_tablet_manager.cc:1428] T 918f5c329e8c43a08c0d72b9e8463c29 P f6fe375836fc4922aecfa3904368afc1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 07:10:03.762773 7767 tablet_bootstrap.cc:654] T 918f5c329e8c43a08c0d72b9e8463c29 P f74dc4545b8e4f7abc5ed2aa102c5cc5: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:03.762787 7464 heartbeater.cc:503] Master 127.0.98.62:35621 was elected leader, sending a full tablet report...
I20260924 07:10:03.763715 7767 log.cc:824] T 918f5c329e8c43a08c0d72b9e8463c29 P f74dc4545b8e4f7abc5ed2aa102c5cc5: Log is configured to *not* fsync() on all Append() calls
I20260924 07:10:03.764577 7767 tablet_bootstrap.cc:492] T 918f5c329e8c43a08c0d72b9e8463c29 P f74dc4545b8e4f7abc5ed2aa102c5cc5: No bootstrap required, opened a new log
I20260924 07:10:03.764628 7767 ts_tablet_manager.cc:1397] T 918f5c329e8c43a08c0d72b9e8463c29 P f74dc4545b8e4f7abc5ed2aa102c5cc5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 07:10:03.766928 7767 raft_consensus.cc:348] T 918f5c329e8c43a08c0d72b9e8463c29 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801287c2cafb4ad786ac93285bbe5080" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43105 } } peers { permanent_uuid: "f6fe375836fc4922aecfa3904368afc1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45969 } } peers { permanent_uuid: "f74dc4545b8e4f7abc5ed2aa102c5cc5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40657 } }
I20260924 07:10:03.767050 7767 raft_consensus.cc:374] T 918f5c329e8c43a08c0d72b9e8463c29 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:03.767072 7767 raft_consensus.cc:729] T 918f5c329e8c43a08c0d72b9e8463c29 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74dc4545b8e4f7abc5ed2aa102c5cc5, State: Initialized, Role: FOLLOWER
I20260924 07:10:03.767143 7767 consensus_queue.cc:260] T 918f5c329e8c43a08c0d72b9e8463c29 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801287c2cafb4ad786ac93285bbe5080" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43105 } } peers { permanent_uuid: "f6fe375836fc4922aecfa3904368afc1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45969 } } peers { permanent_uuid: "f74dc4545b8e4f7abc5ed2aa102c5cc5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40657 } }
I20260924 07:10:03.767318 7767 ts_tablet_manager.cc:1428] T 918f5c329e8c43a08c0d72b9e8463c29 P f74dc4545b8e4f7abc5ed2aa102c5cc5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 07:10:03.767427 7601 heartbeater.cc:503] Master 127.0.98.62:35621 was elected leader, sending a full tablet report...
I20260924 07:10:03.787372 7770 raft_consensus.cc:482] T 918f5c329e8c43a08c0d72b9e8463c29 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:10:03.787469 7770 raft_consensus.cc:504] T 918f5c329e8c43a08c0d72b9e8463c29 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801287c2cafb4ad786ac93285bbe5080" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43105 } } peers { permanent_uuid: "f6fe375836fc4922aecfa3904368afc1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45969 } } peers { permanent_uuid: "f74dc4545b8e4f7abc5ed2aa102c5cc5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40657 } }
I20260924 07:10:03.787942 7770 leader_election.cc:288] T 918f5c329e8c43a08c0d72b9e8463c29 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 801287c2cafb4ad786ac93285bbe5080 (127.0.98.3:43105), f6fe375836fc4922aecfa3904368afc1 (127.0.98.1:45969)
I20260924 07:10:03.791292 7604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.788134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43105 (local address 127.0.98.2:52629)
0924 07:10:03.788331 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:03.788339 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:03.788376 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:03.788515 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:03.788516 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:03.788517 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:10:03.788594 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.788595 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.789360 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.789361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.790308 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.790310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.791035 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.791036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.791111 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.791115 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:10:03.791178 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 07:10:03.791217 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":154}
I20260924 07:10:03.791415 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.788208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50457 (local address 127.0.98.1:45969)
0924 07:10:03.788263 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:03.788265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:03.788276 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:10:03.788302 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:03.788302 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:03.788313 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:03.788373 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:10:03.788585 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.789149 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.790498 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.791062 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.791150 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.791159 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:03.791315 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 07:10:03.791353 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 07:10:03.791424 7605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.788134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45969 (local address 127.0.98.2:50457)
0924 07:10:03.788178 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:03.788190 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:03.788236 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:03.788408 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:03.788410 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:03.788411 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:10:03.788487 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.788489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.789252 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.789253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.790413 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.790416 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.791148 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.791149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.791233 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.791236 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:10:03.791307 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 07:10:03.791348 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 07:10:03.791481 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.788276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52629 (local address 127.0.98.3:43105)
0924 07:10:03.788331 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:03.788333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:03.788371 (+ 38us) server_negotiation.cc:407] Connection header received
0924 07:10:03.788425 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:03.788426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:03.788438 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:03.788506 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:10:03.788670 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.789289 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.790399 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.790949 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.791024 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.791036 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:03.791377 (+ 341us) server_negotiation.cc:299] Negotiation successful
0924 07:10:03.791423 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 07:10:03.791746 7417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "918f5c329e8c43a08c0d72b9e8463c29" candidate_uuid: "f74dc4545b8e4f7abc5ed2aa102c5cc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6fe375836fc4922aecfa3904368afc1" is_pre_election: true
I20260924 07:10:03.791741 7690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "918f5c329e8c43a08c0d72b9e8463c29" candidate_uuid: "f74dc4545b8e4f7abc5ed2aa102c5cc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "801287c2cafb4ad786ac93285bbe5080" is_pre_election: true
I20260924 07:10:03.791898 7690 raft_consensus.cc:2454] T 918f5c329e8c43a08c0d72b9e8463c29 P 801287c2cafb4ad786ac93285bbe5080 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74dc4545b8e4f7abc5ed2aa102c5cc5 in term 0.
I20260924 07:10:03.791905 7417 raft_consensus.cc:2454] T 918f5c329e8c43a08c0d72b9e8463c29 P f6fe375836fc4922aecfa3904368afc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74dc4545b8e4f7abc5ed2aa102c5cc5 in term 0.
I20260924 07:10:03.792093 7489 leader_election.cc:302] T 918f5c329e8c43a08c0d72b9e8463c29 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 801287c2cafb4ad786ac93285bbe5080, f74dc4545b8e4f7abc5ed2aa102c5cc5; no voters:
I20260924 07:10:03.792230 7770 raft_consensus.cc:2781] T 918f5c329e8c43a08c0d72b9e8463c29 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:10:03.792287 7770 raft_consensus.cc:482] T 918f5c329e8c43a08c0d72b9e8463c29 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:10:03.792309 7770 raft_consensus.cc:3037] T 918f5c329e8c43a08c0d72b9e8463c29 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:03.792785 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.789487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37159 (local address 127.0.0.1:57490)
0924 07:10:03.789534 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:03.789543 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:03.789577 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:03.789886 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:03.789886 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:03.789888 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:10:03.789930 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.789932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.790727 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.790728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.791719 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:03.791722 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.792538 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.792539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:03.792595 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.792598 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:10:03.792649 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 07:10:03.792683 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 07:10:03.792790 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.789669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57490 (local address 127.0.98.60:37159)
0924 07:10:03.789733 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:03.789735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:03.789743 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:10:03.789767 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:03.789768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:03.789789 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:03.789859 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:10:03.790004 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.790647 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.791807 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.792388 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.792454 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.792519 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:03.792667 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 07:10:03.792705 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 07:10:03.793120 7294 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260924 07:10:03.793227 7770 raft_consensus.cc:504] T 918f5c329e8c43a08c0d72b9e8463c29 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801287c2cafb4ad786ac93285bbe5080" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43105 } } peers { permanent_uuid: "f6fe375836fc4922aecfa3904368afc1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45969 } } peers { permanent_uuid: "f74dc4545b8e4f7abc5ed2aa102c5cc5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40657 } }
I20260924 07:10:03.793440 7770 leader_election.cc:288] T 918f5c329e8c43a08c0d72b9e8463c29 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [CANDIDATE]: Term 1 election: Requested vote from peers 801287c2cafb4ad786ac93285bbe5080 (127.0.98.3:43105), f6fe375836fc4922aecfa3904368afc1 (127.0.98.1:45969)
I20260924 07:10:03.793498 7690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "918f5c329e8c43a08c0d72b9e8463c29" candidate_uuid: "f74dc4545b8e4f7abc5ed2aa102c5cc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "801287c2cafb4ad786ac93285bbe5080"
I20260924 07:10:03.793572 7690 raft_consensus.cc:3037] T 918f5c329e8c43a08c0d72b9e8463c29 P 801287c2cafb4ad786ac93285bbe5080 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:03.794091 7323 sys_catalog.cc:455] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0556c444f6d740be8b98241bb6a3ebeb. Latest consensus state: current_term: 1 leader_uuid: "0556c444f6d740be8b98241bb6a3ebeb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37159 } attrs { promote: true } } }
I20260924 07:10:03.794199 7323 sys_catalog.cc:458] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [sys.catalog]: This master's current role is: LEARNER
I20260924 07:10:03.794386 7417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "918f5c329e8c43a08c0d72b9e8463c29" candidate_uuid: "f74dc4545b8e4f7abc5ed2aa102c5cc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6fe375836fc4922aecfa3904368afc1"
I20260924 07:10:03.794432 7690 raft_consensus.cc:2454] T 918f5c329e8c43a08c0d72b9e8463c29 P 801287c2cafb4ad786ac93285bbe5080 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74dc4545b8e4f7abc5ed2aa102c5cc5 in term 1.
I20260924 07:10:03.794466 7417 raft_consensus.cc:3037] T 918f5c329e8c43a08c0d72b9e8463c29 P f6fe375836fc4922aecfa3904368afc1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:03.794581 7489 leader_election.cc:302] T 918f5c329e8c43a08c0d72b9e8463c29 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 801287c2cafb4ad786ac93285bbe5080, f74dc4545b8e4f7abc5ed2aa102c5cc5; no voters:
I20260924 07:10:03.794667 7770 raft_consensus.cc:2781] T 918f5c329e8c43a08c0d72b9e8463c29 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:10:03.794811 7770 raft_consensus.cc:686] T 918f5c329e8c43a08c0d72b9e8463c29 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [term 1 LEADER]: Becoming Leader. State: Replica: f74dc4545b8e4f7abc5ed2aa102c5cc5, State: Running, Role: LEADER
I20260924 07:10:03.794898 7770 consensus_queue.cc:237] T 918f5c329e8c43a08c0d72b9e8463c29 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801287c2cafb4ad786ac93285bbe5080" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43105 } } peers { permanent_uuid: "f6fe375836fc4922aecfa3904368afc1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45969 } } peers { permanent_uuid: "f74dc4545b8e4f7abc5ed2aa102c5cc5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40657 } }
I20260924 07:10:03.795379 7417 raft_consensus.cc:2454] T 918f5c329e8c43a08c0d72b9e8463c29 P f6fe375836fc4922aecfa3904368afc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74dc4545b8e4f7abc5ed2aa102c5cc5 in term 1.
I20260924 07:10:03.796003 6499 catalog_manager.cc:5495] T 918f5c329e8c43a08c0d72b9e8463c29 P f74dc4545b8e4f7abc5ed2aa102c5cc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f74dc4545b8e4f7abc5ed2aa102c5cc5 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f74dc4545b8e4f7abc5ed2aa102c5cc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "801287c2cafb4ad786ac93285bbe5080" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6fe375836fc4922aecfa3904368afc1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74dc4545b8e4f7abc5ed2aa102c5cc5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40657 } health_report { overall_health: HEALTHY } } }
I20260924 07:10:03.796540 7059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0556c444f6d740be8b98241bb6a3ebeb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 37159 } attrs { promote: true } } }
I20260924 07:10:03.796655 7059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:03.797118 7060 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb: attempting to promote NON_VOTER f57386635f2c41e28f6b57747c1e62af to VOTER
I20260924 07:10:03.797232 7060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37159 } attrs { promote: false } }
I20260924 07:10:03.797580 7294 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [term 1 LEARNER]: Refusing update from remote peer 0556c444f6d740be8b98241bb6a3ebeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260924 07:10:03.797609 7109 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [term 1 FOLLOWER]: Refusing update from remote peer 0556c444f6d740be8b98241bb6a3ebeb: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260924 07:10:03.797761 7061 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37159 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260924 07:10:03.797844 7059 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260924 07:10:03.798377 7059 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, f57386635f2c41e28f6b57747c1e62af (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37159 } attrs { promote: false } } }
I20260924 07:10:03.798514 7109 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, f57386635f2c41e28f6b57747c1e62af (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37159 } attrs { promote: false } } }
I20260924 07:10:03.798589 7294 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, f57386635f2c41e28f6b57747c1e62af (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37159 } attrs { promote: false } } }
I20260924 07:10:03.799263 7059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0556c444f6d740be8b98241bb6a3ebeb" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37159 } attrs { promote: false } } }
I20260924 07:10:03.799352 7059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:03.799415 7248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0556c444f6d740be8b98241bb6a3ebeb" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37159 } attrs { promote: false } } }
I20260924 07:10:03.799431 7059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0556c444f6d740be8b98241bb6a3ebeb" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37159 } attrs { promote: false } } }
I20260924 07:10:03.799494 7059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:03.799492 7248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:10:03.799499 7323 sys_catalog.cc:455] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0556c444f6d740be8b98241bb6a3ebeb" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37159 } attrs { promote: false } } }
I20260924 07:10:03.799551 7059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0556c444f6d740be8b98241bb6a3ebeb" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37159 } attrs { promote: false } } }
I20260924 07:10:03.799582 7323 sys_catalog.cc:458] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:10:03.799610 7059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:03.811547 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.809455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34346 (local address 127.0.98.2:40657)
0924 07:10:03.809515 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:03.809517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:03.809526 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:10:03.809603 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:03.809604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:03.809619 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:03.809677 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:10:03.809791 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.810256 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.810936 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.811124 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.811173 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.811314 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:10:03.811398 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:10:03.811453 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:03.811485 (+ 32us) server_negotiation.cc:299] Negotiation successful
0924 07:10:03.811499 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 07:10:03.813158 7690 raft_consensus.cc:1261] T 918f5c329e8c43a08c0d72b9e8463c29 P 801287c2cafb4ad786ac93285bbe5080 [term 1 FOLLOWER]: Refusing update from remote peer f74dc4545b8e4f7abc5ed2aa102c5cc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:10:03.813207 7417 raft_consensus.cc:1261] T 918f5c329e8c43a08c0d72b9e8463c29 P f6fe375836fc4922aecfa3904368afc1 [term 1 FOLLOWER]: Refusing update from remote peer f74dc4545b8e4f7abc5ed2aa102c5cc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:10:03.813462 7775 consensus_queue.cc:1035] T 918f5c329e8c43a08c0d72b9e8463c29 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "801287c2cafb4ad786ac93285bbe5080" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:10:03.813582 7770 consensus_queue.cc:1035] T 918f5c329e8c43a08c0d72b9e8463c29 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6fe375836fc4922aecfa3904368afc1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:10:03.816634 7778 mvcc.cc:205] Tried to move back new op lower bound from 7332797660416114688 to 7332797660344377344. Current Snapshot: MvccSnapshot[applied={T|T < 7332797660416114688}]
I20260924 07:10:03.817222 7780 mvcc.cc:205] Tried to move back new op lower bound from 7332797660416114688 to 7332797660344377344. Current Snapshot: MvccSnapshot[applied={T|T < 7332797660416114688 or (T in {7332797660416114688})}]
I20260924 07:10:03.819558 6499 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:10:03.819658 6499 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:10:03.820863 6499 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0556c444f6d740be8b98241bb6a3ebeb: Sending DeleteTablet for 3 replicas of tablet 918f5c329e8c43a08c0d72b9e8463c29
I20260924 07:10:03.821153 7534 tablet_service.cc:1514] Processing DeleteTablet for tablet 918f5c329e8c43a08c0d72b9e8463c29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:10:03 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34334
I20260924 07:10:03.821285 7397 tablet_service.cc:1514] Processing DeleteTablet for tablet 918f5c329e8c43a08c0d72b9e8463c29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:10:03 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59224
I20260924 07:10:03.821696 7670 tablet_service.cc:1514] Processing DeleteTablet for tablet 918f5c329e8c43a08c0d72b9e8463c29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:10:03 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60978
I20260924 07:10:03.821920 7790 tablet_replica.cc:330] T 918f5c329e8c43a08c0d72b9e8463c29 P 801287c2cafb4ad786ac93285bbe5080: stopping tablet replica
I20260924 07:10:03.822049 7790 raft_consensus.cc:2229] T 918f5c329e8c43a08c0d72b9e8463c29 P 801287c2cafb4ad786ac93285bbe5080 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:10:03.822192 7790 raft_consensus.cc:2258] T 918f5c329e8c43a08c0d72b9e8463c29 P 801287c2cafb4ad786ac93285bbe5080 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:10:03.822249 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 6468
I20260924 07:10:03.823660 7790 ts_tablet_manager.cc:1905] T 918f5c329e8c43a08c0d72b9e8463c29 P 801287c2cafb4ad786ac93285bbe5080: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:10:03.825654 7790 ts_tablet_manager.cc:1918] T 918f5c329e8c43a08c0d72b9e8463c29 P 801287c2cafb4ad786ac93285bbe5080: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:10:03.825784 7790 log.cc:1196] T 918f5c329e8c43a08c0d72b9e8463c29 P 801287c2cafb4ad786ac93285bbe5080: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-2/wal/wals/918f5c329e8c43a08c0d72b9e8463c29
I20260924 07:10:03.826100 7790 ts_tablet_manager.cc:1939] T 918f5c329e8c43a08c0d72b9e8463c29 P 801287c2cafb4ad786ac93285bbe5080: Deleting consensus metadata
W20260924 07:10:03.828790 7626 connection.cc:535] server connection from 127.0.0.1:60978 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 07:10:03.829453 7788 tablet_replica.cc:330] T 918f5c329e8c43a08c0d72b9e8463c29 P f6fe375836fc4922aecfa3904368afc1: stopping tablet replica
I20260924 07:10:03.829560 7788 raft_consensus.cc:2229] T 918f5c329e8c43a08c0d72b9e8463c29 P f6fe375836fc4922aecfa3904368afc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:10:03.829655 7788 raft_consensus.cc:2258] T 918f5c329e8c43a08c0d72b9e8463c29 P f6fe375836fc4922aecfa3904368afc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:10:03.829762 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.821660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57496 (local address 127.0.98.60:37159)
0924 07:10:03.821697 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:03.821699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:03.821964 (+ 265us) server_negotiation.cc:407] Connection header received
0924 07:10:03.822063 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:03.822064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:03.822074 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:03.822103 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:03.822982 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.823736 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.824806 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.825106 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.825149 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.826788 (+ 1639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:03.826789 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:03.826789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:03.826796 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:03.828506 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.828916 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.828917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.828918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.828972 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.829172 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.829173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.829173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.829294 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:03.829366 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:03.829593 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 07:10:03.829670 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 07:10:03.830148 7788 ts_tablet_manager.cc:1905] T 918f5c329e8c43a08c0d72b9e8463c29 P f6fe375836fc4922aecfa3904368afc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:10:03.831938 7788 ts_tablet_manager.cc:1918] T 918f5c329e8c43a08c0d72b9e8463c29 P f6fe375836fc4922aecfa3904368afc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:10:03.832020 7788 log.cc:1196] T 918f5c329e8c43a08c0d72b9e8463c29 P f6fe375836fc4922aecfa3904368afc1: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-0/wal/wals/918f5c329e8c43a08c0d72b9e8463c29
I20260924 07:10:03.832342 7788 ts_tablet_manager.cc:1939] T 918f5c329e8c43a08c0d72b9e8463c29 P f6fe375836fc4922aecfa3904368afc1: Deleting consensus metadata
W20260924 07:10:03.832690 7351 connection.cc:441] server connection from 127.0.0.1:59224 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59224 (request call id 1) could send its response
I20260924 07:10:03.832880 7787 tablet_replica.cc:330] T 918f5c329e8c43a08c0d72b9e8463c29 P f74dc4545b8e4f7abc5ed2aa102c5cc5: stopping tablet replica
I20260924 07:10:03.832978 7787 raft_consensus.cc:2229] T 918f5c329e8c43a08c0d72b9e8463c29 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:10:03.833105 7787 raft_consensus.cc:2258] T 918f5c329e8c43a08c0d72b9e8463c29 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:10:03.833535 7787 ts_tablet_manager.cc:1905] T 918f5c329e8c43a08c0d72b9e8463c29 P f74dc4545b8e4f7abc5ed2aa102c5cc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:10:03.835114 7787 ts_tablet_manager.cc:1918] T 918f5c329e8c43a08c0d72b9e8463c29 P f74dc4545b8e4f7abc5ed2aa102c5cc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:10:03.835173 7787 log.cc:1196] T 918f5c329e8c43a08c0d72b9e8463c29 P f74dc4545b8e4f7abc5ed2aa102c5cc5: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-1/wal/wals/918f5c329e8c43a08c0d72b9e8463c29
I20260924 07:10:03.835426 7787 ts_tablet_manager.cc:1939] T 918f5c329e8c43a08c0d72b9e8463c29 P f74dc4545b8e4f7abc5ed2aa102c5cc5: Deleting consensus metadata
W20260924 07:10:03.835767 7487 connection.cc:441] server connection from 127.0.0.1:34334 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34334 (request call id 1) could send its response
I20260924 07:10:03.838833 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.830678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57504 (local address 127.0.98.60:37159)
0924 07:10:03.830718 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:03.830720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:03.830831 (+ 111us) server_negotiation.cc:407] Connection header received
0924 07:10:03.830874 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:03.830875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:03.830884 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:03.830939 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:03.831632 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.832345 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.833438 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.833739 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.833778 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.835348 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:03.835349 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:03.835349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:03.835356 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:03.837597 (+ 2241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.837971 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.837972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.837972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.838029 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.838249 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.838250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.838250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.838388 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:03.838488 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:03.838656 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 07:10:03.838730 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 07:10:03.839568 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:03.831018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42598 (local address 127.0.98.61:38773)
0924 07:10:03.831058 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:03.831060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:03.831294 (+ 234us) server_negotiation.cc:407] Connection header received
0924 07:10:03.831390 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:03.831391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:03.831403 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:03.831431 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:03.833152 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.833852 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.834932 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:03.835273 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:03.835320 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:03.836719 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:03.836720 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:03.836720 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:03.836728 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:03.838321 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.838728 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.838729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.838729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.838783 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:03.838981 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:03.838981 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:03.838982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:03.839099 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:03.839164 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:03.839355 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 07:10:03.839422 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 07:10:04.817660 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:04.817387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.98.1:41299)
0924 07:10:04.817511 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:04.817598 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":69,"threads_started":1}
I20260924 07:10:04.817986 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:04.817450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.98.2:54069)
0924 07:10:04.817637 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:04.817708 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":55,"threads_started":1}
W20260924 07:10:04.818058 7464 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35621 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
W20260924 07:10:04.818156 7601 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35621 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
I20260924 07:10:04.818670 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:04.818410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.98.3:42301)
0924 07:10:04.818557 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:04.818626 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":51,"threads_started":1}
W20260924 07:10:04.818832 7737 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35621 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
I20260924 07:10:05.321509 7800 raft_consensus.cc:482] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0556c444f6d740be8b98241bb6a3ebeb)
I20260924 07:10:05.321645 7800 raft_consensus.cc:504] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37159 } attrs { promote: false } }
I20260924 07:10:05.321892 7800 leader_election.cc:288] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0556c444f6d740be8b98241bb6a3ebeb (127.0.98.62:35621), 45abe618fb274f6db2ea422007f6c684 (127.0.98.61:38773)
I20260924 07:10:05.322060 7801 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0556c444f6d740be8b98241bb6a3ebeb)
I20260924 07:10:05.322171 7801 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37159 } attrs { promote: false } }
I20260924 07:10:05.322220 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:05.321996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.0.1:41958)
0924 07:10:05.322112 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:05.322171 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":45,"threads_started":1}
W20260924 07:10:05.322361 7271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
I20260924 07:10:05.322664 7801 leader_election.cc:288] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0556c444f6d740be8b98241bb6a3ebeb (127.0.98.62:35621), f57386635f2c41e28f6b57747c1e62af (127.0.98.60:37159)
I20260924 07:10:05.322747 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:05.322666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.0.1:41968)
0924 07:10:05.322705 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:05.322724 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 07:10:05.322777 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:05.322545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.0.1:41966)
0924 07:10:05.322651 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:05.322736 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":44,"threads_started":1}
W20260924 07:10:05.322862 7086 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
I20260924 07:10:05.323669 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:05.323594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.0.1:41982)
0924 07:10:05.323623 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:05.323644 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260924 07:10:05.323680 7271 leader_election.cc:334] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0556c444f6d740be8b98241bb6a3ebeb (127.0.98.62:35621): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
W20260924 07:10:05.323762 7086 leader_election.cc:334] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0556c444f6d740be8b98241bb6a3ebeb (127.0.98.62:35621): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
I20260924 07:10:05.326436 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:05.322594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37159 (local address 127.0.0.1:57508)
0924 07:10:05.322766 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:05.322776 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:05.322883 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:05.323139 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:05.323140 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:05.323142 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:10:05.323187 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:05.323191 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:05.324155 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:05.324156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:05.325399 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:05.325401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:05.326170 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:05.326171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:05.326233 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:05.326237 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:10:05.326298 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 07:10:05.326338 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":40,"threads_started":1}
I20260924 07:10:05.326442 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:05.322011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38773 (local address 127.0.0.1:42612)
0924 07:10:05.322196 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:05.322218 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:05.322288 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:05.323082 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:05.323083 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:05.323086 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:10:05.323154 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:05.323156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:05.324095 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:05.324096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:05.325399 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:05.325401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:05.326170 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:05.326171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:05.326249 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:05.326253 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:10:05.326306 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 07:10:05.326348 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":2,"thread_start_us":55,"threads_started":1}
I20260924 07:10:05.326655 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:05.322094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42612 (local address 127.0.98.61:38773)
0924 07:10:05.322256 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:05.322258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:05.322957 (+ 699us) server_negotiation.cc:407] Connection header received
0924 07:10:05.322981 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:05.322982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:05.322991 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:05.323218 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:10:05.323246 (+ 28us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:05.324017 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:05.325514 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:05.326079 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:05.326162 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:05.326173 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:05.326336 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 07:10:05.326376 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":45,"threads_started":1}
I20260924 07:10:05.326654 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:05.322769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57508 (local address 127.0.98.60:37159)
0924 07:10:05.322905 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:05.322907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:05.322915 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:10:05.322989 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:05.322990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:05.323001 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:05.323088 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:10:05.323260 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:05.324079 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:05.325512 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:05.326084 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:05.326162 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:05.326173 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:05.326331 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 07:10:05.326376 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
I20260924 07:10:05.326823 7109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f57386635f2c41e28f6b57747c1e62af" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "45abe618fb274f6db2ea422007f6c684" is_pre_election: true
I20260924 07:10:05.326841 7294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45abe618fb274f6db2ea422007f6c684" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f57386635f2c41e28f6b57747c1e62af" is_pre_election: true
I20260924 07:10:05.326927 7294 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45abe618fb274f6db2ea422007f6c684 in term 1.
I20260924 07:10:05.326927 7109 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f57386635f2c41e28f6b57747c1e62af in term 1.
I20260924 07:10:05.327119 7269 leader_election.cc:302] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45abe618fb274f6db2ea422007f6c684, f57386635f2c41e28f6b57747c1e62af; no voters: 0556c444f6d740be8b98241bb6a3ebeb
I20260924 07:10:05.327127 7085 leader_election.cc:302] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45abe618fb274f6db2ea422007f6c684, f57386635f2c41e28f6b57747c1e62af; no voters: 0556c444f6d740be8b98241bb6a3ebeb
I20260924 07:10:05.327234 7800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 07:10:05.327253 7801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 07:10:05.327278 7800 raft_consensus.cc:482] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0556c444f6d740be8b98241bb6a3ebeb)
I20260924 07:10:05.327291 7801 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0556c444f6d740be8b98241bb6a3ebeb)
I20260924 07:10:05.327301 7800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [term 1 FOLLOWER]: Advancing to term 2
I20260924 07:10:05.327315 7801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [term 1 FOLLOWER]: Advancing to term 2
I20260924 07:10:05.328016 7800 raft_consensus.cc:504] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37159 } attrs { promote: false } }
I20260924 07:10:05.328042 7801 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37159 } attrs { promote: false } }
I20260924 07:10:05.328152 7800 leader_election.cc:288] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [CANDIDATE]: Term 2 election: Requested vote from peers 0556c444f6d740be8b98241bb6a3ebeb (127.0.98.62:35621), 45abe618fb274f6db2ea422007f6c684 (127.0.98.61:38773)
I20260924 07:10:05.328171 7801 leader_election.cc:288] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [CANDIDATE]: Term 2 election: Requested vote from peers 0556c444f6d740be8b98241bb6a3ebeb (127.0.98.62:35621), f57386635f2c41e28f6b57747c1e62af (127.0.98.60:37159)
I20260924 07:10:05.328317 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:05.328227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.0.1:41996)
0924 07:10:05.328263 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:05.328292 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260924 07:10:05.328364 7294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45abe618fb274f6db2ea422007f6c684" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f57386635f2c41e28f6b57747c1e62af"
I20260924 07:10:05.328374 7109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f57386635f2c41e28f6b57747c1e62af" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "45abe618fb274f6db2ea422007f6c684"
I20260924 07:10:05.328444 7294 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 45abe618fb274f6db2ea422007f6c684 in current term 2: Already voted for candidate f57386635f2c41e28f6b57747c1e62af in this term.
I20260924 07:10:05.328459 7109 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f57386635f2c41e28f6b57747c1e62af in current term 2: Already voted for candidate 45abe618fb274f6db2ea422007f6c684 in this term.
I20260924 07:10:05.328687 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:05.328296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.0.1:42012)
0924 07:10:05.328347 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:05.328662 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 07:10:05.328881 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:05.328769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.0.1:42024)
0924 07:10:05.328809 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:05.328860 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 07:10:05.328997 7271 leader_election.cc:334] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0556c444f6d740be8b98241bb6a3ebeb (127.0.98.62:35621): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
I20260924 07:10:05.329046 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:05.328946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.0.1:42028)
0924 07:10:05.329001 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:05.329024 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 07:10:05.329051 7271 leader_election.cc:302] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f57386635f2c41e28f6b57747c1e62af; no voters: 0556c444f6d740be8b98241bb6a3ebeb, 45abe618fb274f6db2ea422007f6c684
W20260924 07:10:05.329087 7086 leader_election.cc:334] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0556c444f6d740be8b98241bb6a3ebeb (127.0.98.62:35621): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
I20260924 07:10:05.329123 7086 leader_election.cc:302] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45abe618fb274f6db2ea422007f6c684; no voters: 0556c444f6d740be8b98241bb6a3ebeb, f57386635f2c41e28f6b57747c1e62af
I20260924 07:10:05.329147 7800 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 07:10:05.329303 7801 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 07:10:05.818791 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:05.818520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.98.2:50569)
0924 07:10:05.818693 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:05.818735 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":95,"threads_started":1}
I20260924 07:10:05.818827 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:05.818442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.98.1:59985)
0924 07:10:05.818758 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:05.818788 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":97,"threads_started":1}
I20260924 07:10:05.819322 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:05.819153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.98.3:54895)
0924 07:10:05.819253 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:05.819289 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":61,"threads_started":1}
I20260924 07:10:06.819427 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:06.819200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.98.2:57431)
0924 07:10:06.819328 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:06.819371 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":75,"threads_started":1}
I20260924 07:10:06.819690 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:06.819447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.98.1:36391)
0924 07:10:06.819610 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:06.819647 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":52,"threads_started":1}
I20260924 07:10:06.819936 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:06.819729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.98.3:55579)
0924 07:10:06.819861 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:06.819896 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":47,"threads_started":1}
I20260924 07:10:06.866452 7816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:10:06.866549 7816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37159 } attrs { promote: false } }
I20260924 07:10:06.866735 7816 leader_election.cc:288] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0556c444f6d740be8b98241bb6a3ebeb (127.0.98.62:35621), f57386635f2c41e28f6b57747c1e62af (127.0.98.60:37159)
I20260924 07:10:06.867056 7294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45abe618fb274f6db2ea422007f6c684" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f57386635f2c41e28f6b57747c1e62af" is_pre_election: true
I20260924 07:10:06.867108 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:06.866897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.0.1:42256)
0924 07:10:06.867021 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:06.867067 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":59,"threads_started":1}
I20260924 07:10:06.867163 7294 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45abe618fb274f6db2ea422007f6c684 in term 2.
I20260924 07:10:06.867355 7085 leader_election.cc:302] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45abe618fb274f6db2ea422007f6c684, f57386635f2c41e28f6b57747c1e62af; no voters:
I20260924 07:10:06.867473 7816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 07:10:06.867532 7816 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:10:06.867568 7816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [term 2 FOLLOWER]: Advancing to term 3
I20260924 07:10:06.867614 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:06.867496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.0.1:42270)
0924 07:10:06.867538 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:06.867569 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260924 07:10:06.867724 7086 leader_election.cc:334] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0556c444f6d740be8b98241bb6a3ebeb (127.0.98.62:35621): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
I20260924 07:10:06.868381 7816 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37159 } attrs { promote: false } }
I20260924 07:10:06.868526 7816 leader_election.cc:288] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [CANDIDATE]: Term 3 election: Requested vote from peers 0556c444f6d740be8b98241bb6a3ebeb (127.0.98.62:35621), f57386635f2c41e28f6b57747c1e62af (127.0.98.60:37159)
I20260924 07:10:06.868728 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:06.868599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.0.1:42280)
0924 07:10:06.868672 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:06.868700 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 07:10:06.868783 7294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "45abe618fb274f6db2ea422007f6c684" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f57386635f2c41e28f6b57747c1e62af"
I20260924 07:10:06.868866 7294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [term 2 FOLLOWER]: Advancing to term 3
I20260924 07:10:06.869177 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:06.869020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.0.1:42286)
0924 07:10:06.869105 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:06.869157 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260924 07:10:06.869283 7086 leader_election.cc:334] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 0556c444f6d740be8b98241bb6a3ebeb (127.0.98.62:35621): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
I20260924 07:10:06.869614 7294 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45abe618fb274f6db2ea422007f6c684 in term 3.
I20260924 07:10:06.869880 7085 leader_election.cc:302] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45abe618fb274f6db2ea422007f6c684, f57386635f2c41e28f6b57747c1e62af; no voters: 0556c444f6d740be8b98241bb6a3ebeb
I20260924 07:10:06.870048 7816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 07:10:06.870187 7816 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [term 3 LEADER]: Becoming Leader. State: Replica: 45abe618fb274f6db2ea422007f6c684, State: Running, Role: LEADER
I20260924 07:10:06.870286 7816 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37159 } attrs { promote: false } }
I20260924 07:10:06.870653 7819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45abe618fb274f6db2ea422007f6c684. Latest consensus state: current_term: 3 leader_uuid: "45abe618fb274f6db2ea422007f6c684" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37159 } attrs { promote: false } } }
I20260924 07:10:06.870767 7819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:06.870951 7821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:10:06.871347 7821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:10:06.871407 7821 catalog_manager.cc:1259] Loaded cluster ID: c6d426aa15ca4806959a3a3058567fa2
I20260924 07:10:06.871421 7821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:10:06.871559 7821 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:10:06.871610 7821 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684: Loaded TSK: 0
I20260924 07:10:06.871706 7821 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:10:06.926905 7098 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42598:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:10:06.928574 7098 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:10:06.929750 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:06.929626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.0.1:42292)
0924 07:10:06.929686 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:06.929719 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 07:10:06.929742 7294 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [term 3 FOLLOWER]: Refusing update from remote peer 45abe618fb274f6db2ea422007f6c684: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260924 07:10:06.930020 7816 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37159 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260924 07:10:06.930105 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:06.929992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.0.1:42308)
0924 07:10:06.930046 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:06.930080 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 07:10:06.930222 7086 consensus_peers.cc:487] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 -> Peer 0556c444f6d740be8b98241bb6a3ebeb (127.0.98.62:35621): Couldn't send request to peer 0556c444f6d740be8b98241bb6a3ebeb. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 07:10:06.930683 7823 sys_catalog.cc:455] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45abe618fb274f6db2ea422007f6c684. Latest consensus state: current_term: 3 leader_uuid: "45abe618fb274f6db2ea422007f6c684" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37159 } attrs { promote: false } } }
I20260924 07:10:06.930794 7823 sys_catalog.cc:458] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:10:06.931180 7816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "45abe618fb274f6db2ea422007f6c684" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37159 } attrs { promote: false } } }
I20260924 07:10:06.931277 7816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684 [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:06.931285 7823 sys_catalog.cc:455] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "45abe618fb274f6db2ea422007f6c684" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0556c444f6d740be8b98241bb6a3ebeb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35621 } } peers { permanent_uuid: "45abe618fb274f6db2ea422007f6c684" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38773 } attrs { promote: false } } peers { permanent_uuid: "f57386635f2c41e28f6b57747c1e62af" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37159 } attrs { promote: false } } }
I20260924 07:10:06.931360 7823 sys_catalog.cc:458] T 00000000000000000000000000000000 P f57386635f2c41e28f6b57747c1e62af [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:10:06.936961 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:06.933171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45969 (local address 127.0.0.1:59234)
0924 07:10:06.933332 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:06.933343 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:06.933401 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:06.933712 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:06.933713 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:06.933715 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:10:06.933748 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:06.933749 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:06.934631 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:06.934631 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:06.935878 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:06.935881 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:06.936689 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:06.936690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:06.936794 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:06.936799 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:10:06.936830 (+ 31us) client_negotiation.cc:240] Negotiation successful
0924 07:10:06.936876 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":48,"thread_start_us":79,"threads_started":1}
I20260924 07:10:06.937059 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:06.933171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43105 (local address 127.0.0.1:60986)
0924 07:10:06.933202 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:06.933213 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:06.933271 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:06.933920 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:06.933920 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:06.933922 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:10:06.933965 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:06.933966 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:06.934858 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:06.934859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:06.935914 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:06.935916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:06.936647 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:06.936648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:06.936706 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:06.936711 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:10:06.936895 (+ 184us) client_negotiation.cc:240] Negotiation successful
0924 07:10:06.936945 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260924 07:10:06.937220 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:06.933314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59234 (local address 127.0.98.1:45969)
0924 07:10:06.933594 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:06.933596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:06.933605 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:10:06.933630 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:06.933631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:06.933641 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:06.933713 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:10:06.933808 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:06.934519 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:06.935970 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:06.936555 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:06.937078 (+ 523us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:06.937093 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:06.937111 (+ 18us) server_negotiation.cc:299] Negotiation successful
0924 07:10:06.937152 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":53,"threads_started":1}
I20260924 07:10:06.937722 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:06.933364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60986 (local address 127.0.98.3:43105)
0924 07:10:06.933785 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:06.933787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:06.933794 (+ 7us) server_negotiation.cc:407] Connection header received
0924 07:10:06.933818 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:06.933819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:06.933828 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:06.934020 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:10:06.934084 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:06.934792 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:06.935992 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:06.936555 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:06.936640 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:06.936658 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:06.936994 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 07:10:06.937658 (+ 664us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":51,"threads_started":1}
I20260924 07:10:06.937793 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:06.933232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40657 (local address 127.0.0.1:34358)
0924 07:10:06.933414 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:06.933422 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:06.933463 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:06.933641 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:06.933642 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:06.933643 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:10:06.933696 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:06.933697 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:06.934634 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:06.934634 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:06.936693 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:06.936694 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:06.937535 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:06.937536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:06.937602 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:06.937606 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:10:06.937690 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 07:10:06.937729 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":2,"thread_start_us":79,"threads_started":1}
I20260924 07:10:06.937955 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:06.933414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34358 (local address 127.0.98.2:40657)
0924 07:10:06.933518 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:06.933520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:06.933528 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:10:06.933554 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:06.933555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:06.933567 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:06.933639 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:10:06.933757 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:06.934552 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:06.936927 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:06.937449 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:06.937535 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:06.937546 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:06.937698 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 07:10:06.937739 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":54,"threads_started":1}
I20260924 07:10:06.937999 7397 tablet_service.cc:1467] Processing CreateTablet for tablet eba99546bb814cb7b7c2a6b889b80cf8 (DEFAULT_TABLE table=test-table [id=19fe42fa3c7242af80c265d738919429]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:10:06.938140 7397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eba99546bb814cb7b7c2a6b889b80cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:06.938216 7534 tablet_service.cc:1467] Processing CreateTablet for tablet eba99546bb814cb7b7c2a6b889b80cf8 (DEFAULT_TABLE table=test-table [id=19fe42fa3c7242af80c265d738919429]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:10:06.938354 7534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eba99546bb814cb7b7c2a6b889b80cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:06.939432 7833 tablet_bootstrap.cc:492] T eba99546bb814cb7b7c2a6b889b80cf8 P f6fe375836fc4922aecfa3904368afc1: Bootstrap starting.
I20260924 07:10:06.939522 7834 tablet_bootstrap.cc:492] T eba99546bb814cb7b7c2a6b889b80cf8 P f74dc4545b8e4f7abc5ed2aa102c5cc5: Bootstrap starting.
I20260924 07:10:06.939663 7670 tablet_service.cc:1467] Processing CreateTablet for tablet eba99546bb814cb7b7c2a6b889b80cf8 (DEFAULT_TABLE table=test-table [id=19fe42fa3c7242af80c265d738919429]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:10:06.939782 7670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eba99546bb814cb7b7c2a6b889b80cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:06.939914 7833 tablet_bootstrap.cc:654] T eba99546bb814cb7b7c2a6b889b80cf8 P f6fe375836fc4922aecfa3904368afc1: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:06.939939 7834 tablet_bootstrap.cc:654] T eba99546bb814cb7b7c2a6b889b80cf8 P f74dc4545b8e4f7abc5ed2aa102c5cc5: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:06.940573 7833 tablet_bootstrap.cc:492] T eba99546bb814cb7b7c2a6b889b80cf8 P f6fe375836fc4922aecfa3904368afc1: No bootstrap required, opened a new log
I20260924 07:10:06.940629 7833 ts_tablet_manager.cc:1397] T eba99546bb814cb7b7c2a6b889b80cf8 P f6fe375836fc4922aecfa3904368afc1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:10:06.940814 7833 raft_consensus.cc:348] T eba99546bb814cb7b7c2a6b889b80cf8 P f6fe375836fc4922aecfa3904368afc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6fe375836fc4922aecfa3904368afc1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45969 } } peers { permanent_uuid: "801287c2cafb4ad786ac93285bbe5080" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43105 } } peers { permanent_uuid: "f74dc4545b8e4f7abc5ed2aa102c5cc5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40657 } }
I20260924 07:10:06.940891 7833 raft_consensus.cc:374] T eba99546bb814cb7b7c2a6b889b80cf8 P f6fe375836fc4922aecfa3904368afc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:06.940910 7833 raft_consensus.cc:729] T eba99546bb814cb7b7c2a6b889b80cf8 P f6fe375836fc4922aecfa3904368afc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6fe375836fc4922aecfa3904368afc1, State: Initialized, Role: FOLLOWER
I20260924 07:10:06.940955 7833 consensus_queue.cc:260] T eba99546bb814cb7b7c2a6b889b80cf8 P f6fe375836fc4922aecfa3904368afc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6fe375836fc4922aecfa3904368afc1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45969 } } peers { permanent_uuid: "801287c2cafb4ad786ac93285bbe5080" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43105 } } peers { permanent_uuid: "f74dc4545b8e4f7abc5ed2aa102c5cc5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40657 } }
I20260924 07:10:06.941108 7834 tablet_bootstrap.cc:492] T eba99546bb814cb7b7c2a6b889b80cf8 P f74dc4545b8e4f7abc5ed2aa102c5cc5: No bootstrap required, opened a new log
I20260924 07:10:06.941128 7833 ts_tablet_manager.cc:1428] T eba99546bb814cb7b7c2a6b889b80cf8 P f6fe375836fc4922aecfa3904368afc1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 07:10:06.941210 7834 ts_tablet_manager.cc:1397] T eba99546bb814cb7b7c2a6b889b80cf8 P f74dc4545b8e4f7abc5ed2aa102c5cc5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:10:06.941351 7834 raft_consensus.cc:348] T eba99546bb814cb7b7c2a6b889b80cf8 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6fe375836fc4922aecfa3904368afc1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45969 } } peers { permanent_uuid: "801287c2cafb4ad786ac93285bbe5080" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43105 } } peers { permanent_uuid: "f74dc4545b8e4f7abc5ed2aa102c5cc5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40657 } }
I20260924 07:10:06.941470 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:06.941350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.98.1:59197)
0924 07:10:06.941407 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:06.941441 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 07:10:06.941465 7834 raft_consensus.cc:374] T eba99546bb814cb7b7c2a6b889b80cf8 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:06.941572 7834 raft_consensus.cc:729] T eba99546bb814cb7b7c2a6b889b80cf8 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f74dc4545b8e4f7abc5ed2aa102c5cc5, State: Initialized, Role: FOLLOWER
I20260924 07:10:06.941663 7834 consensus_queue.cc:260] T eba99546bb814cb7b7c2a6b889b80cf8 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6fe375836fc4922aecfa3904368afc1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45969 } } peers { permanent_uuid: "801287c2cafb4ad786ac93285bbe5080" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43105 } } peers { permanent_uuid: "f74dc4545b8e4f7abc5ed2aa102c5cc5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40657 } }
I20260924 07:10:06.941876 7834 ts_tablet_manager.cc:1428] T eba99546bb814cb7b7c2a6b889b80cf8 P f74dc4545b8e4f7abc5ed2aa102c5cc5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 07:10:06.941957 7838 tablet_bootstrap.cc:492] T eba99546bb814cb7b7c2a6b889b80cf8 P 801287c2cafb4ad786ac93285bbe5080: Bootstrap starting.
I20260924 07:10:06.942204 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:06.942099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.98.2:56771)
0924 07:10:06.942151 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:06.942180 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 07:10:06.942310 7838 tablet_bootstrap.cc:654] T eba99546bb814cb7b7c2a6b889b80cf8 P 801287c2cafb4ad786ac93285bbe5080: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:06.942633 7602 heartbeater.cc:503] Master 127.0.98.61:38773 was elected leader, sending a full tablet report...
I20260924 07:10:06.942926 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:06.942838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.98.2:47523)
0924 07:10:06.942875 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:06.942901 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 07:10:06.943048 7838 tablet_bootstrap.cc:492] T eba99546bb814cb7b7c2a6b889b80cf8 P 801287c2cafb4ad786ac93285bbe5080: No bootstrap required, opened a new log
I20260924 07:10:06.943106 7838 ts_tablet_manager.cc:1397] T eba99546bb814cb7b7c2a6b889b80cf8 P 801287c2cafb4ad786ac93285bbe5080: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:10:06.943267 7838 raft_consensus.cc:348] T eba99546bb814cb7b7c2a6b889b80cf8 P 801287c2cafb4ad786ac93285bbe5080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6fe375836fc4922aecfa3904368afc1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45969 } } peers { permanent_uuid: "801287c2cafb4ad786ac93285bbe5080" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43105 } } peers { permanent_uuid: "f74dc4545b8e4f7abc5ed2aa102c5cc5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40657 } }
I20260924 07:10:06.943318 7838 raft_consensus.cc:374] T eba99546bb814cb7b7c2a6b889b80cf8 P 801287c2cafb4ad786ac93285bbe5080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:06.943361 7838 raft_consensus.cc:729] T eba99546bb814cb7b7c2a6b889b80cf8 P 801287c2cafb4ad786ac93285bbe5080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 801287c2cafb4ad786ac93285bbe5080, State: Initialized, Role: FOLLOWER
I20260924 07:10:06.943428 7838 consensus_queue.cc:260] T eba99546bb814cb7b7c2a6b889b80cf8 P 801287c2cafb4ad786ac93285bbe5080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6fe375836fc4922aecfa3904368afc1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45969 } } peers { permanent_uuid: "801287c2cafb4ad786ac93285bbe5080" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43105 } } peers { permanent_uuid: "f74dc4545b8e4f7abc5ed2aa102c5cc5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40657 } }
I20260924 07:10:06.943647 7838 ts_tablet_manager.cc:1428] T eba99546bb814cb7b7c2a6b889b80cf8 P 801287c2cafb4ad786ac93285bbe5080: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:06.943918 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:06.943814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.98.3:55873)
0924 07:10:06.943867 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:06.943895 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 07:10:06.944255 7738 heartbeater.cc:503] Master 127.0.98.61:38773 was elected leader, sending a full tablet report...
I20260924 07:10:06.944433 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:06.944343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.98.3:43779)
0924 07:10:06.944378 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:06.944401 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260924 07:10:06.977720 7739 tablet.cc:2369] T eba99546bb814cb7b7c2a6b889b80cf8 P 801287c2cafb4ad786ac93285bbe5080: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:10:07.079600 7603 tablet.cc:2369] T eba99546bb814cb7b7c2a6b889b80cf8 P f74dc4545b8e4f7abc5ed2aa102c5cc5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:10:07.168893 7839 raft_consensus.cc:482] T eba99546bb814cb7b7c2a6b889b80cf8 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:10:07.169037 7839 raft_consensus.cc:504] T eba99546bb814cb7b7c2a6b889b80cf8 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6fe375836fc4922aecfa3904368afc1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45969 } } peers { permanent_uuid: "801287c2cafb4ad786ac93285bbe5080" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43105 } } peers { permanent_uuid: "f74dc4545b8e4f7abc5ed2aa102c5cc5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40657 } }
I20260924 07:10:07.169297 7839 leader_election.cc:288] T eba99546bb814cb7b7c2a6b889b80cf8 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6fe375836fc4922aecfa3904368afc1 (127.0.98.1:45969), 801287c2cafb4ad786ac93285bbe5080 (127.0.98.3:43105)
I20260924 07:10:07.169502 7690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba99546bb814cb7b7c2a6b889b80cf8" candidate_uuid: "f74dc4545b8e4f7abc5ed2aa102c5cc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "801287c2cafb4ad786ac93285bbe5080" is_pre_election: true
I20260924 07:10:07.169585 7690 raft_consensus.cc:2454] T eba99546bb814cb7b7c2a6b889b80cf8 P 801287c2cafb4ad786ac93285bbe5080 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74dc4545b8e4f7abc5ed2aa102c5cc5 in term 0.
I20260924 07:10:07.169597 7417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba99546bb814cb7b7c2a6b889b80cf8" candidate_uuid: "f74dc4545b8e4f7abc5ed2aa102c5cc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6fe375836fc4922aecfa3904368afc1" is_pre_election: true
I20260924 07:10:07.169747 7417 raft_consensus.cc:2454] T eba99546bb814cb7b7c2a6b889b80cf8 P f6fe375836fc4922aecfa3904368afc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f74dc4545b8e4f7abc5ed2aa102c5cc5 in term 0.
I20260924 07:10:07.169809 7489 leader_election.cc:302] T eba99546bb814cb7b7c2a6b889b80cf8 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 801287c2cafb4ad786ac93285bbe5080, f74dc4545b8e4f7abc5ed2aa102c5cc5; no voters:
I20260924 07:10:07.169905 7839 raft_consensus.cc:2781] T eba99546bb814cb7b7c2a6b889b80cf8 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:10:07.169955 7839 raft_consensus.cc:482] T eba99546bb814cb7b7c2a6b889b80cf8 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:10:07.169987 7839 raft_consensus.cc:3037] T eba99546bb814cb7b7c2a6b889b80cf8 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:07.170704 7839 raft_consensus.cc:504] T eba99546bb814cb7b7c2a6b889b80cf8 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6fe375836fc4922aecfa3904368afc1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45969 } } peers { permanent_uuid: "801287c2cafb4ad786ac93285bbe5080" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43105 } } peers { permanent_uuid: "f74dc4545b8e4f7abc5ed2aa102c5cc5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40657 } }
I20260924 07:10:07.170845 7839 leader_election.cc:288] T eba99546bb814cb7b7c2a6b889b80cf8 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [CANDIDATE]: Term 1 election: Requested vote from peers f6fe375836fc4922aecfa3904368afc1 (127.0.98.1:45969), 801287c2cafb4ad786ac93285bbe5080 (127.0.98.3:43105)
I20260924 07:10:07.171018 7417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba99546bb814cb7b7c2a6b889b80cf8" candidate_uuid: "f74dc4545b8e4f7abc5ed2aa102c5cc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6fe375836fc4922aecfa3904368afc1"
I20260924 07:10:07.171074 7690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eba99546bb814cb7b7c2a6b889b80cf8" candidate_uuid: "f74dc4545b8e4f7abc5ed2aa102c5cc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "801287c2cafb4ad786ac93285bbe5080"
I20260924 07:10:07.171119 7417 raft_consensus.cc:3037] T eba99546bb814cb7b7c2a6b889b80cf8 P f6fe375836fc4922aecfa3904368afc1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:07.171159 7690 raft_consensus.cc:3037] T eba99546bb814cb7b7c2a6b889b80cf8 P 801287c2cafb4ad786ac93285bbe5080 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:07.171749 7690 raft_consensus.cc:2454] T eba99546bb814cb7b7c2a6b889b80cf8 P 801287c2cafb4ad786ac93285bbe5080 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74dc4545b8e4f7abc5ed2aa102c5cc5 in term 1.
I20260924 07:10:07.171835 7417 raft_consensus.cc:2454] T eba99546bb814cb7b7c2a6b889b80cf8 P f6fe375836fc4922aecfa3904368afc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f74dc4545b8e4f7abc5ed2aa102c5cc5 in term 1.
I20260924 07:10:07.171916 7489 leader_election.cc:302] T eba99546bb814cb7b7c2a6b889b80cf8 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 801287c2cafb4ad786ac93285bbe5080, f74dc4545b8e4f7abc5ed2aa102c5cc5; no voters:
I20260924 07:10:07.172029 7839 raft_consensus.cc:2781] T eba99546bb814cb7b7c2a6b889b80cf8 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:10:07.172148 7839 raft_consensus.cc:686] T eba99546bb814cb7b7c2a6b889b80cf8 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [term 1 LEADER]: Becoming Leader. State: Replica: f74dc4545b8e4f7abc5ed2aa102c5cc5, State: Running, Role: LEADER
I20260924 07:10:07.172189 7839 consensus_queue.cc:237] T eba99546bb814cb7b7c2a6b889b80cf8 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6fe375836fc4922aecfa3904368afc1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45969 } } peers { permanent_uuid: "801287c2cafb4ad786ac93285bbe5080" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43105 } } peers { permanent_uuid: "f74dc4545b8e4f7abc5ed2aa102c5cc5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40657 } }
I20260924 07:10:07.172626 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:07.172514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.98.2:36475)
0924 07:10:07.172554 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:07.172590 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260924 07:10:07.173054 7098 catalog_manager.cc:5495] T eba99546bb814cb7b7c2a6b889b80cf8 P f74dc4545b8e4f7abc5ed2aa102c5cc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f74dc4545b8e4f7abc5ed2aa102c5cc5 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f74dc4545b8e4f7abc5ed2aa102c5cc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6fe375836fc4922aecfa3904368afc1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "801287c2cafb4ad786ac93285bbe5080" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f74dc4545b8e4f7abc5ed2aa102c5cc5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40657 } health_report { overall_health: HEALTHY } } }
W20260924 07:10:07.190430 7466 tablet.cc:2369] T eba99546bb814cb7b7c2a6b889b80cf8 P f6fe375836fc4922aecfa3904368afc1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:10:07.223289 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:07.221137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34372 (local address 127.0.98.2:40657)
0924 07:10:07.221197 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:07.221199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:07.221207 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:10:07.221231 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:07.221232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:07.221243 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:07.221288 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:10:07.221405 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:07.221879 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:07.222643 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:07.222844 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:07.222894 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:07.223022 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:10:07.223044 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:10:07.223083 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:07.223214 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 07:10:07.223230 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 07:10:07.224269 7417 raft_consensus.cc:1261] T eba99546bb814cb7b7c2a6b889b80cf8 P f6fe375836fc4922aecfa3904368afc1 [term 1 FOLLOWER]: Refusing update from remote peer f74dc4545b8e4f7abc5ed2aa102c5cc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:10:07.224267 7690 raft_consensus.cc:1261] T eba99546bb814cb7b7c2a6b889b80cf8 P 801287c2cafb4ad786ac93285bbe5080 [term 1 FOLLOWER]: Refusing update from remote peer f74dc4545b8e4f7abc5ed2aa102c5cc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:10:07.224640 7842 consensus_queue.cc:1035] T eba99546bb814cb7b7c2a6b889b80cf8 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6fe375836fc4922aecfa3904368afc1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:10:07.224807 7839 consensus_queue.cc:1035] T eba99546bb814cb7b7c2a6b889b80cf8 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "801287c2cafb4ad786ac93285bbe5080" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:10:07.225389 7465 heartbeater.cc:503] Master 127.0.98.61:38773 was elected leader, sending a full tablet report...
I20260924 07:10:07.225625 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:07.225480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.98.1:43053)
0924 07:10:07.225524 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:07.225600 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 07:10:07.225620 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:07.225517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.98.3:48361)
0924 07:10:07.225556 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:07.225586 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 07:10:07.226905 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:07.226756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.98.2:48461)
0924 07:10:07.226852 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:07.226880 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 07:10:07.227180 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:07.227083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.98.3:51797)
0924 07:10:07.227124 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:07.227152 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 07:10:07.227334 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:07.227247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.98.2:45205)
0924 07:10:07.227279 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:07.227304 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 07:10:07.228231 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:07.228145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35621 (local address 127.0.98.1:48275)
0924 07:10:07.228174 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:07.228201 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260924 07:10:07.228861 7098 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:10:07.228951 7098 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:10:07.230247 7098 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 45abe618fb274f6db2ea422007f6c684: Sending DeleteTablet for 3 replicas of tablet eba99546bb814cb7b7c2a6b889b80cf8
I20260924 07:10:07.230489 7397 tablet_service.cc:1514] Processing DeleteTablet for tablet eba99546bb814cb7b7c2a6b889b80cf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:10:07 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:59234
I20260924 07:10:07.230562 7670 tablet_service.cc:1514] Processing DeleteTablet for tablet eba99546bb814cb7b7c2a6b889b80cf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:10:07 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:60986
I20260924 07:10:07.230582 7534 tablet_service.cc:1514] Processing DeleteTablet for tablet eba99546bb814cb7b7c2a6b889b80cf8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:10:07 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:34358
I20260924 07:10:07.230731 7856 tablet_replica.cc:330] T eba99546bb814cb7b7c2a6b889b80cf8 P f74dc4545b8e4f7abc5ed2aa102c5cc5: stopping tablet replica
I20260924 07:10:07.230731 7855 tablet_replica.cc:330] T eba99546bb814cb7b7c2a6b889b80cf8 P f6fe375836fc4922aecfa3904368afc1: stopping tablet replica
I20260924 07:10:07.230777 7856 raft_consensus.cc:2229] T eba99546bb814cb7b7c2a6b889b80cf8 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:10:07.230782 7855 raft_consensus.cc:2229] T eba99546bb814cb7b7c2a6b889b80cf8 P f6fe375836fc4922aecfa3904368afc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:10:07.230821 7855 raft_consensus.cc:2258] T eba99546bb814cb7b7c2a6b889b80cf8 P f6fe375836fc4922aecfa3904368afc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:10:07.230832 7856 raft_consensus.cc:2258] T eba99546bb814cb7b7c2a6b889b80cf8 P f74dc4545b8e4f7abc5ed2aa102c5cc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:10:07.230985 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 7246
I20260924 07:10:07.231096 7857 tablet_replica.cc:330] T eba99546bb814cb7b7c2a6b889b80cf8 P 801287c2cafb4ad786ac93285bbe5080: stopping tablet replica
I20260924 07:10:07.231182 7857 raft_consensus.cc:2229] T eba99546bb814cb7b7c2a6b889b80cf8 P 801287c2cafb4ad786ac93285bbe5080 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:10:07.231603 7857 raft_consensus.cc:2258] T eba99546bb814cb7b7c2a6b889b80cf8 P 801287c2cafb4ad786ac93285bbe5080 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:10:07.231936 7857 ts_tablet_manager.cc:1905] T eba99546bb814cb7b7c2a6b889b80cf8 P 801287c2cafb4ad786ac93285bbe5080: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:10:07.233487 7856 ts_tablet_manager.cc:1905] T eba99546bb814cb7b7c2a6b889b80cf8 P f74dc4545b8e4f7abc5ed2aa102c5cc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:10:07.233845 7857 ts_tablet_manager.cc:1918] T eba99546bb814cb7b7c2a6b889b80cf8 P 801287c2cafb4ad786ac93285bbe5080: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:10:07.233955 7857 log.cc:1196] T eba99546bb814cb7b7c2a6b889b80cf8 P 801287c2cafb4ad786ac93285bbe5080: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-2/wal/wals/eba99546bb814cb7b7c2a6b889b80cf8
I20260924 07:10:07.234266 7857 ts_tablet_manager.cc:1939] T eba99546bb814cb7b7c2a6b889b80cf8 P 801287c2cafb4ad786ac93285bbe5080: Deleting consensus metadata
I20260924 07:10:07.234961 7085 catalog_manager.cc:4841] TS 801287c2cafb4ad786ac93285bbe5080 (127.0.98.3:43105): tablet eba99546bb814cb7b7c2a6b889b80cf8 (table test-table [id=19fe42fa3c7242af80c265d738919429]) successfully deleted
I20260924 07:10:07.235204 7856 ts_tablet_manager.cc:1918] T eba99546bb814cb7b7c2a6b889b80cf8 P f74dc4545b8e4f7abc5ed2aa102c5cc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:10:07.235266 7856 log.cc:1196] T eba99546bb814cb7b7c2a6b889b80cf8 P f74dc4545b8e4f7abc5ed2aa102c5cc5: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790233772407728-392-0/minicluster-data/ts-1/wal/wals/eba99546bb814cb7b7c2a6b889b80cf8
I20260924 07:10:07.235539 7856 ts_tablet_manager.cc:1939] T eba99546bb814cb7b7c2a6b889b80cf8 P f74dc4545b8e4f7abc5ed2aa102c5cc5: Deleting consensus metadata
I20260924 07:10:07.235872 7083 catalog_manager.cc:4841] TS f74dc4545b8e4f7abc5ed2aa102c5cc5 (127.0.98.2:40657): tablet eba99546bb814cb7b7c2a6b889b80cf8 (table test-table [id=19fe42fa3c7242af80c265d738919429]) successfully deleted
I20260924 07:10:07.237005 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 7471
I20260924 07:10:07.237031 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:07.236910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45969 (local address 127.0.0.1:59238)
0924 07:10:07.236964 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:07.236998 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45969: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 07:10:07.237600 7086 catalog_manager.cc:4584] TS f6fe375836fc4922aecfa3904368afc1 (127.0.98.1:45969): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eba99546bb814cb7b7c2a6b889b80cf8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45969: connect: Connection refused (error 111)
I20260924 07:10:07.243937 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 7607
I20260924 07:10:07.248586 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 6964
I20260924 07:10:07.253361 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 7153
2026-09-24T07:10:07Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6261 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:10:07 dist-test-slave-14ww krb5kdc[7860](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:10:07 dist-test-slave-14ww krb5kdc[7860](info): set up 2 sockets
Sep 24 07:10:07 dist-test-slave-14ww krb5kdc[7860](info): commencing operation
krb5kdc: starting...
W20260924 07:10:08.293260 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:10:08 dist-test-slave-14ww krb5kdc[7860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:10:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:10:08Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 07:10:08.411432 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34025
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:08.490676 7873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:08.490868 7873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:08.490890 7873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:08.491957 7873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:10:08.491988 7873 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 07:10:08.492002 7873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:08.492014 7873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:08.492026 7873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:10:08.492038 7873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:10:08.493448 7873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34025
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:10:08.493634 7873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:08.493795 7873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:08.498591 7879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:08.498799 7878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:08.498934 7873 server_base.cc:1034] running on GCE node
W20260924 07:10:08.499006 7881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:08.499253 7873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:08.499536 7873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:08.500824 7873 hybrid_clock.cc:648] HybridClock initialized: now 1790233808500811 us; error 30 us; skew 500 ppm
Sep 24 07:10:08 dist-test-slave-14ww krb5kdc[7860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233808, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:08.503278 7873 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 07:10:08.503726 7873 webserver.cc:460] Webserver started at http://127.0.98.62:39191/ using document root <none> and password file <none>
I20260924 07:10:08.503962 7873 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:08.504006 7873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:08.504145 7873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:08.505225 7873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "32fa0114c29249baacf9a200be709192"
format_stamp: "Formatted at 2026-09-24 07:10:08 on dist-test-slave-14ww"
server_key: "cc33e5f0f6aeb16ac41488d6aa59bd97"
server_key_iv: "f8335f426e86c0cb2ad4e6d04816a4a0"
server_key_version: "encryptionkey@0"
I20260924 07:10:08.505568 7873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "32fa0114c29249baacf9a200be709192"
format_stamp: "Formatted at 2026-09-24 07:10:08 on dist-test-slave-14ww"
server_key: "cc33e5f0f6aeb16ac41488d6aa59bd97"
server_key_iv: "f8335f426e86c0cb2ad4e6d04816a4a0"
server_key_version: "encryptionkey@0"
I20260924 07:10:08.506935 7873 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:10:08.507740 7888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:08.508003 7873 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:10:08.508121 7873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "32fa0114c29249baacf9a200be709192"
format_stamp: "Formatted at 2026-09-24 07:10:08 on dist-test-slave-14ww"
server_key: "cc33e5f0f6aeb16ac41488d6aa59bd97"
server_key_iv: "f8335f426e86c0cb2ad4e6d04816a4a0"
server_key_version: "encryptionkey@0"
I20260924 07:10:08.508210 7873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:08.526762 7873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:08.528965 7873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:08.529141 7873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:08.533682 7873 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34025
I20260924 07:10:08.533726 7950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34025 every 8 connection(s)
I20260924 07:10:08.534049 7873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:10:08.534632 7951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:08.535741 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 7873
I20260924 07:10:08.535856 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:10:08.536055 392 external_mini_cluster.cc:1428] Setting key e619cfdadc849b40ee3ea2fc807397bd
I20260924 07:10:08.537331 7951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32fa0114c29249baacf9a200be709192: Bootstrap starting.
I20260924 07:10:08.537889 7951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32fa0114c29249baacf9a200be709192: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:08.538165 7951 log.cc:824] T 00000000000000000000000000000000 P 32fa0114c29249baacf9a200be709192: Log is configured to *not* fsync() on all Append() calls
I20260924 07:10:08.538872 7951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32fa0114c29249baacf9a200be709192: No bootstrap required, opened a new log
I20260924 07:10:08.540575 7951 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32fa0114c29249baacf9a200be709192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32fa0114c29249baacf9a200be709192" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34025 } }
I20260924 07:10:08.540722 7951 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32fa0114c29249baacf9a200be709192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:08.540800 7951 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32fa0114c29249baacf9a200be709192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32fa0114c29249baacf9a200be709192, State: Initialized, Role: FOLLOWER
I20260924 07:10:08.540928 7951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32fa0114c29249baacf9a200be709192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32fa0114c29249baacf9a200be709192" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34025 } }
I20260924 07:10:08.541018 7951 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32fa0114c29249baacf9a200be709192 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:10:08.541052 7951 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32fa0114c29249baacf9a200be709192 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:10:08.541102 7951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32fa0114c29249baacf9a200be709192 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:08.541582 7951 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32fa0114c29249baacf9a200be709192 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32fa0114c29249baacf9a200be709192" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34025 } }
I20260924 07:10:08.541652 7951 leader_election.cc:302] T 00000000000000000000000000000000 P 32fa0114c29249baacf9a200be709192 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32fa0114c29249baacf9a200be709192; no voters:
I20260924 07:10:08.541767 7951 leader_election.cc:288] T 00000000000000000000000000000000 P 32fa0114c29249baacf9a200be709192 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:10:08.541885 7956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32fa0114c29249baacf9a200be709192 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:10:08.542155 7956 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32fa0114c29249baacf9a200be709192 [term 1 LEADER]: Becoming Leader. State: Replica: 32fa0114c29249baacf9a200be709192, State: Running, Role: LEADER
I20260924 07:10:08.542210 7951 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32fa0114c29249baacf9a200be709192 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:10:08.542250 7956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32fa0114c29249baacf9a200be709192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32fa0114c29249baacf9a200be709192" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34025 } }
I20260924 07:10:08.542565 7956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32fa0114c29249baacf9a200be709192 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32fa0114c29249baacf9a200be709192. Latest consensus state: current_term: 1 leader_uuid: "32fa0114c29249baacf9a200be709192" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32fa0114c29249baacf9a200be709192" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34025 } } }
I20260924 07:10:08.542624 7956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32fa0114c29249baacf9a200be709192 [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:08.542588 7957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32fa0114c29249baacf9a200be709192 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32fa0114c29249baacf9a200be709192" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32fa0114c29249baacf9a200be709192" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34025 } } }
I20260924 07:10:08.542680 7957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32fa0114c29249baacf9a200be709192 [sys.catalog]: This master's current role is: LEADER
Sep 24 07:10:08 dist-test-slave-14ww krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 07:10:08.543126 7961 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:10:08.543938 7961 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:10:08.546916 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:08.536577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33454 (local address 127.0.98.62:34025)
0924 07:10:08.536781 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:08.536796 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:08.536828 (+ 32us) server_negotiation.cc:407] Connection header received
0924 07:10:08.537244 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:08.537251 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:08.537296 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:08.537449 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:08.538141 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:08.539282 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:08.540035 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:08.540448 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:08.540577 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:08.543320 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:08.543325 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:08.543326 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:08.543338 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:08.545344 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:08.545704 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:08.545714 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:08.545715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:08.545829 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:08.546013 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:08.546014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:08.546014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:08.546301 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:08.546445 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:08.546537 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 07:10:08.546673 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":82,"threads_started":1}
I20260924 07:10:08.551824 7961 catalog_manager.cc:1347] Generated new cluster ID: f36d70a353214076b950a47ea653dd78
I20260924 07:10:08.551883 7961 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:10:08.565160 7961 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:10:08.565729 7961 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:10:08.568379 7961 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32fa0114c29249baacf9a200be709192: Generated new TSK 0
I20260924 07:10:08.568565 7961 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 07:10:08.608938 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34025
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:08.706260 7977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:08.706449 7977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:08.706472 7977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:08.707619 7977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:08.707666 7977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:08.707687 7977 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:10:08.707742 7977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:10:08.709180 7977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:10:08.709414 7977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:08.709652 7977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:08.714582 7985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:08.714843 7977 server_base.cc:1034] running on GCE node
W20260924 07:10:08.714949 7982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:08.715552 7983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:08.715804 7977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:08.716012 7977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:08.717183 7977 hybrid_clock.cc:648] HybridClock initialized: now 1790233808717167 us; error 28 us; skew 500 ppm
Sep 24 07:10:08 dist-test-slave-14ww krb5kdc[7860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233808, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:08.719877 7977 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 07:10:08.720356 7977 webserver.cc:460] Webserver started at http://127.0.98.1:36221/ using document root <none> and password file <none>
I20260924 07:10:08.720649 7977 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:08.720757 7977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:08.720921 7977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:08.722015 7977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-0/data/instance:
uuid: "83906483277942bab937e411fc4c666e"
format_stamp: "Formatted at 2026-09-24 07:10:08 on dist-test-slave-14ww"
server_key: "d44b0ae7ab68d2752fa581a07967f247"
server_key_iv: "3916d392b7bc20e90ed91795ec153e27"
server_key_version: "encryptionkey@0"
I20260924 07:10:08.722389 7977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-0/wal/instance:
uuid: "83906483277942bab937e411fc4c666e"
format_stamp: "Formatted at 2026-09-24 07:10:08 on dist-test-slave-14ww"
server_key: "d44b0ae7ab68d2752fa581a07967f247"
server_key_iv: "3916d392b7bc20e90ed91795ec153e27"
server_key_version: "encryptionkey@0"
I20260924 07:10:08.723768 7977 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:10:08.724527 7992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:08.724774 7977 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:10:08.724891 7977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-0/wal
uuid: "83906483277942bab937e411fc4c666e"
format_stamp: "Formatted at 2026-09-24 07:10:08 on dist-test-slave-14ww"
server_key: "d44b0ae7ab68d2752fa581a07967f247"
server_key_iv: "3916d392b7bc20e90ed91795ec153e27"
server_key_version: "encryptionkey@0"
I20260924 07:10:08.724979 7977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:08.742010 7977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:08.744764 7977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:08.744937 7977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:08.745458 7977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:10:08.745522 7977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:08.745559 7977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:10:08.745599 7977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 07:10:08 dist-test-slave-14ww krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233808, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 07:10:08.753559 7977 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45771
I20260924 07:10:08.753907 7977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:10:08.753892 8107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45771 every 8 connection(s)
I20260924 07:10:08.754751 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 7977
I20260924 07:10:08.754865 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:10:08.755084 392 external_mini_cluster.cc:1428] Setting key fe6120cd8142f85f058fab8a534dd86d
I20260924 07:10:08.756786 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:08.746089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57957 (local address 127.0.98.62:34025)
0924 07:10:08.746149 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:08.746151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:08.746376 (+ 225us) server_negotiation.cc:407] Connection header received
0924 07:10:08.746893 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:08.746895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:08.746913 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:08.746965 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:08.747867 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:08.748827 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:08.750081 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:08.750420 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:08.750490 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:08.753325 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:08.753327 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:08.753327 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:08.753336 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:08.755117 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:08.755619 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:08.755620 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:08.755621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:08.755678 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:08.756095 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:08.756096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:08.756096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:08.756217 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:08.756278 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:08.756592 (+ 314us) server_negotiation.cc:299] Negotiation successful
0924 07:10:08.756661 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 07:10:08.757036 8001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:08.746096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34025 (local address 127.0.98.1:57957)
0924 07:10:08.746259 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:08.746291 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:08.746786 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:08.747041 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:08.747045 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:08.747403 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:08.747728 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:08.747733 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:08.748941 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:08.748942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:08.749990 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:08.749993 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:08.750542 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:08.750543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:08.750727 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:08.751336 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:08.751347 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:08.753231 (+ 1884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:08.755218 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:08.755225 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:08.755227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:08.755529 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:08.755880 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:08.755881 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:08.755881 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:08.756017 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:08.756290 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:08.756296 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:08.756459 (+ 163us) client_negotiation.cc:769] Sending connection context
0924 07:10:08.756569 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 07:10:08.756766 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":102,"threads_started":1}
I20260924 07:10:08.757537 8108 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34025
I20260924 07:10:08.757634 8108 heartbeater.cc:464] Registering TS with master...
I20260924 07:10:08.757812 8108 heartbeater.cc:511] Master 127.0.98.62:34025 requested a full tablet report, sending...
I20260924 07:10:08.758350 7905 ts_manager.cc:194] Registered new tserver with Master: 83906483277942bab937e411fc4c666e (127.0.98.1:45771)
I20260924 07:10:08.759229 7905 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57957
I20260924 07:10:08.764712 8115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:08.758657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34025 (local address 127.0.98.1:32927)
0924 07:10:08.758820 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:08.758833 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:08.758897 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:08.759036 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:08.759036 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:08.759267 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:08.759343 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:08.759345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:08.760227 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:08.760228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:08.761040 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:08.761042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:08.761371 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:08.761372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:08.761449 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:08.761911 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:08.761915 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:08.762167 (+ 252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:08.763694 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:08.763695 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:08.763696 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:08.763862 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:08.764074 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:08.764075 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:08.764075 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:08.764130 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:08.764378 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:08.764379 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:08.764419 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 07:10:08.764508 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 07:10:08.764605 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":84,"threads_started":1}
I20260924 07:10:08.764693 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:08.758817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32927 (local address 127.0.98.62:34025)
0924 07:10:08.758849 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:08.758850 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:08.758882 (+ 32us) server_negotiation.cc:407] Connection header received
0924 07:10:08.758955 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:08.758956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:08.758964 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:08.759019 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:08.759423 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:08.760111 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:08.761113 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:08.761290 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:08.761372 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:08.762290 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:08.762291 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:08.762291 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:08.762298 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:08.763596 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:08.763949 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:08.763950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:08.763951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:08.764001 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:08.764202 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:08.764203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:08.764203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:08.764311 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:08.764385 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:08.764532 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 07:10:08.764603 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 07:10:08.783480 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34025
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:08.862407 8117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:08.862651 8117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:08.862689 8117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:08.863862 8117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:08.863925 8117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:08.863963 8117 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:10:08.864017 8117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:10:08.865495 8117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:10:08.865758 8117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:08.865976 8117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:08.871534 8123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:08.871575 8122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:08.871575 8125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:08.871865 8117 server_base.cc:1034] running on GCE node
I20260924 07:10:08.872071 8117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:08.872323 8117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:08.873564 8117 hybrid_clock.cc:648] HybridClock initialized: now 1790233808873534 us; error 49 us; skew 500 ppm
Sep 24 07:10:08 dist-test-slave-14ww krb5kdc[7860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233808, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:08.876453 8117 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 07:10:08.876957 8117 webserver.cc:460] Webserver started at http://127.0.98.2:40161/ using document root <none> and password file <none>
I20260924 07:10:08.877203 8117 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:08.877260 8117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:08.877352 8117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:08.878449 8117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-1/data/instance:
uuid: "5f1865a617384ee3aa88bc78243d7949"
format_stamp: "Formatted at 2026-09-24 07:10:08 on dist-test-slave-14ww"
server_key: "bfbd776461f997d78ce08a252b1cd23a"
server_key_iv: "79cc99c185d48e1858d56fd46576068a"
server_key_version: "encryptionkey@0"
I20260924 07:10:08.878938 8117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5f1865a617384ee3aa88bc78243d7949"
format_stamp: "Formatted at 2026-09-24 07:10:08 on dist-test-slave-14ww"
server_key: "bfbd776461f997d78ce08a252b1cd23a"
server_key_iv: "79cc99c185d48e1858d56fd46576068a"
server_key_version: "encryptionkey@0"
I20260924 07:10:08.880441 8117 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:10:08.881265 8132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:08.881466 8117 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 07:10:08.881562 8117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-1/wal
uuid: "5f1865a617384ee3aa88bc78243d7949"
format_stamp: "Formatted at 2026-09-24 07:10:08 on dist-test-slave-14ww"
server_key: "bfbd776461f997d78ce08a252b1cd23a"
server_key_iv: "79cc99c185d48e1858d56fd46576068a"
server_key_version: "encryptionkey@0"
I20260924 07:10:08.881636 8117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:08.902032 8117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:08.905154 8117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:08.905313 8117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:08.905876 8117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:10:08.905949 8117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:08.905998 8117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:10:08.906028 8117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:08.913321 8117 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42777
I20260924 07:10:08.913654 8247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42777 every 8 connection(s)
I20260924 07:10:08.913764 8117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 07:10:08 dist-test-slave-14ww krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233808, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 07:10:08.917367 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:08.906458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35975 (local address 127.0.98.62:34025)
0924 07:10:08.906535 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:08.906538 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:08.906745 (+ 207us) server_negotiation.cc:407] Connection header received
0924 07:10:08.907252 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:08.907253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:08.907280 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:08.907364 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:08.908488 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:08.909484 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:08.911422 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:08.911656 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:08.911739 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:08.914554 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:08.914555 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:08.914555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:08.914562 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:08.915890 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:08.916364 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:08.916365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:08.916365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:08.916422 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:08.916683 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:08.916684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:08.916684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:08.916868 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:08.916928 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:08.917222 (+ 294us) server_negotiation.cc:299] Negotiation successful
0924 07:10:08.917304 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:10:08.917488 8141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:08.906427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34025 (local address 127.0.98.2:35975)
0924 07:10:08.906602 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:08.906642 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:08.907148 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:08.907511 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:08.907515 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:08.907898 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:08.908208 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:08.908215 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:08.909783 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:08.909784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:08.911344 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:08.911347 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:08.911751 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:08.911753 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:08.911910 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:08.912595 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:08.912606 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:08.914468 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:08.915983 (+ 1515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:08.915988 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:08.915990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:08.916212 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:08.916510 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:08.916511 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:08.916511 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:08.916616 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:08.916935 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:08.916940 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:08.917060 (+ 120us) client_negotiation.cc:769] Sending connection context
0924 07:10:08.917168 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 07:10:08.917306 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":104,"threads_started":1}
I20260924 07:10:08.917971 8248 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34025
I20260924 07:10:08.918052 8248 heartbeater.cc:464] Registering TS with master...
I20260924 07:10:08.918109 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 8117
I20260924 07:10:08.918244 8248 heartbeater.cc:511] Master 127.0.98.62:34025 requested a full tablet report, sending...
I20260924 07:10:08.918256 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:10:08.918494 392 external_mini_cluster.cc:1428] Setting key 95975d4e4bd3bdfda6caa00f0136f810
I20260924 07:10:08.918893 7905 ts_manager.cc:194] Registered new tserver with Master: 5f1865a617384ee3aa88bc78243d7949 (127.0.98.2:42777)
I20260924 07:10:08.919461 7905 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35975
I20260924 07:10:08.925415 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:08.919122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50519 (local address 127.0.98.62:34025)
0924 07:10:08.919256 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:08.919258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:08.919379 (+ 121us) server_negotiation.cc:407] Connection header received
0924 07:10:08.919465 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:08.919466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:08.919474 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:08.919521 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:08.919890 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:08.920360 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:08.921368 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:08.921558 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:08.921619 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:08.922703 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:08.922705 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:08.922705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:08.922711 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:08.924166 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:08.924591 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:08.924591 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:08.924592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:08.924649 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:08.924893 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:08.924894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:08.924894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:08.925038 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:08.925093 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:08.925286 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 07:10:08.925354 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260924 07:10:08.925488 8254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:08.919057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34025 (local address 127.0.98.2:50519)
0924 07:10:08.919318 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:08.919329 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:08.919405 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:08.919522 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:08.919523 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:08.919736 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:08.919797 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:08.919799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:08.920457 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:08.920458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:08.921242 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:08.921245 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:08.921639 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:08.921639 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:08.921703 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:08.922177 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:08.922182 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:08.922592 (+ 410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:08.924259 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:08.924260 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:08.924261 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:08.924509 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:08.924723 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:08.924724 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:08.924724 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:08.924820 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:08.925109 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:08.925110 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:08.925160 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 07:10:08.925251 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 07:10:08.925369 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 07:10:08.951988 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34025
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:09.031962 8257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:09.032147 8257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:09.032169 8257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:09.033257 8257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:09.033293 8257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:09.033313 8257 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 07:10:09.033330 8257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:10:09.034694 8257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:10:09.034870 8257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:09.035035 8257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:09.039969 8263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:09.039947 8265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:09.039948 8262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:09.040199 8257 server_base.cc:1034] running on GCE node
I20260924 07:10:09.040364 8257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:09.040550 8257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:09.041757 8257 hybrid_clock.cc:648] HybridClock initialized: now 1790233809041731 us; error 56 us; skew 500 ppm
Sep 24 07:10:09 dist-test-slave-14ww krb5kdc[7860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233809, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:09.044409 8257 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 07:10:09.044968 8257 webserver.cc:460] Webserver started at http://127.0.98.3:46583/ using document root <none> and password file <none>
I20260924 07:10:09.045202 8257 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:09.045274 8257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:09.045424 8257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:09.046468 8257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-2/data/instance:
uuid: "3ba1e1457a494fc58f23c23c7e156a59"
format_stamp: "Formatted at 2026-09-24 07:10:09 on dist-test-slave-14ww"
server_key: "29ab67a34299bff28397d2f6619c7a81"
server_key_iv: "981c3a61ea09d96ba70dcda27b16025f"
server_key_version: "encryptionkey@0"
I20260924 07:10:09.046836 8257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3ba1e1457a494fc58f23c23c7e156a59"
format_stamp: "Formatted at 2026-09-24 07:10:09 on dist-test-slave-14ww"
server_key: "29ab67a34299bff28397d2f6619c7a81"
server_key_iv: "981c3a61ea09d96ba70dcda27b16025f"
server_key_version: "encryptionkey@0"
I20260924 07:10:09.048139 8257 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:10:09.048914 8272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:09.049086 8257 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:10:09.049185 8257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-2/wal
uuid: "3ba1e1457a494fc58f23c23c7e156a59"
format_stamp: "Formatted at 2026-09-24 07:10:09 on dist-test-slave-14ww"
server_key: "29ab67a34299bff28397d2f6619c7a81"
server_key_iv: "981c3a61ea09d96ba70dcda27b16025f"
server_key_version: "encryptionkey@0"
I20260924 07:10:09.049268 8257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:09.070464 8257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:09.072862 8257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:09.073015 8257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:09.073611 8257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:10:09.073658 8257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:09.073693 8257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:10:09.073721 8257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 07:10:09 dist-test-slave-14ww krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233809, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 07:10:09.080854 8257 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44545
I20260924 07:10:09.080971 8387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44545 every 8 connection(s)
I20260924 07:10:09.081248 8257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:10:09.083189 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:09.074034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40895 (local address 127.0.98.62:34025)
0924 07:10:09.074097 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:09.074100 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:09.074321 (+ 221us) server_negotiation.cc:407] Connection header received
0924 07:10:09.074811 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:09.074813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:09.074826 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:09.074911 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:09.075712 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.076545 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.077696 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.078033 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.078126 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:09.080376 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:09.080377 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:09.080377 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:09.080385 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:09.082121 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:09.082465 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:09.082466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:09.082466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:09.082514 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:09.082743 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:09.082744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:09.082744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:09.082836 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:09.082884 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:09.083074 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 07:10:09.083129 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 07:10:09.083355 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:09.074030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34025 (local address 127.0.98.3:40895)
0924 07:10:09.074184 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:09.074224 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:09.074706 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:09.074962 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:09.074966 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:09.075332 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:09.075613 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:09.075620 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.076644 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.076645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:09.077631 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:09.077634 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.078174 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.078175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:09.078307 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:09.078842 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:09.078851 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:09.080304 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:09.082192 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:09.082196 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:09.082198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:09.082401 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:09.082598 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:09.082598 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:09.082598 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:09.082686 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:09.082888 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:09.082892 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:09.082990 (+ 98us) client_negotiation.cc:769] Sending connection context
0924 07:10:09.083065 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 07:10:09.083174 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":88,"threads_started":1}
I20260924 07:10:09.083853 8388 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34025
I20260924 07:10:09.083933 8388 heartbeater.cc:464] Registering TS with master...
I20260924 07:10:09.084128 8388 heartbeater.cc:511] Master 127.0.98.62:34025 requested a full tablet report, sending...
I20260924 07:10:09.084564 7904 ts_manager.cc:194] Registered new tserver with Master: 3ba1e1457a494fc58f23c23c7e156a59 (127.0.98.3:44545)
I20260924 07:10:09.085111 7904 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40895
I20260924 07:10:09.086866 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 8257
I20260924 07:10:09.086944 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:10:09.087152 392 external_mini_cluster.cc:1428] Setting key 03814d8968b395d8a9bdf8dc4bb650ab
I20260924 07:10:09.087999 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:10:09.090402 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:09.084929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36809 (local address 127.0.98.62:34025)
0924 07:10:09.084966 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:09.084968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:09.085020 (+ 52us) server_negotiation.cc:407] Connection header received
0924 07:10:09.085084 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:09.085084 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:09.085091 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:09.085133 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:09.085527 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.085983 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.086710 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.086939 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.086997 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:09.088076 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:09.088077 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:09.088077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:09.088084 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:09.089316 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:09.089666 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:09.089667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:09.089668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:09.089725 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:09.089922 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:09.089923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:09.089923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:09.090029 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:09.090103 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:09.090252 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 07:10:09.090316 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 07:10:09.090420 8394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:09.084841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34025 (local address 127.0.98.3:36809)
0924 07:10:09.084965 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:09.084975 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:09.085031 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:09.085138 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:09.085139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:09.085339 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:09.085413 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:09.085414 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.086080 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.086081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:09.086597 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:09.086599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.087044 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.087044 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:09.087102 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:09.087586 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:09.087590 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:09.087961 (+ 371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:09.089413 (+ 1452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:09.089414 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:09.089415 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:09.089579 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:09.089792 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:09.089792 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:09.089793 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:09.089848 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:09.090095 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:09.090095 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:09.090134 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 07:10:09.090222 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 07:10:09.090317 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":77,"threads_started":1}
Sep 24 07:10:09 dist-test-slave-14ww krb5kdc[7860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233809, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260924 07:10:09.103477 7950 acceptor_pool.cc:285] unable to collect diagnostics on RPC socket 127.0.98.62:34025: Invalid argument: socket address is ambiguous
Sep 24 07:10:09 dist-test-slave-14ww krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:09 dist-test-slave-14ww krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 07:10:09.109078 7954 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:09.103617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33458 (local address 127.0.98.62:34025)
0924 07:10:09.103655 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:09.103657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:09.103666 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:10:09.103848 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:09.103849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:09.103858 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:09.103912 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:09.104465 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.105093 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.105815 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.105989 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.106048 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:09.108988 (+ 2940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33458: BlockingRecv error: recv got EOF from 127.0.0.1:33458 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Sep 24 07:10:09 dist-test-slave-14ww krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233809, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 07:10:09.116343 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:09.109809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33474 (local address 127.0.98.62:34025)
0924 07:10:09.109846 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:09.109848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:09.109903 (+ 55us) server_negotiation.cc:407] Connection header received
0924 07:10:09.109977 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:09.109978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:09.109986 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:09.110034 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:09.110555 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.111033 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.111783 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.111966 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.112047 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:09.114066 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:09.114067 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:09.114067 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:09.114075 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:09.115232 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:09.115569 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:09.115570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:09.115571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:09.115628 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:09.115857 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:09.115858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:09.115858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:09.115961 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:09.116058 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:09.116182 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 07:10:09.116250 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 07:10:09.117734 7904 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:10:09.119447 7904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:10:09.125540 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:09.121546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44545 (local address 127.0.0.1:39354)
0924 07:10:09.121829 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:09.121840 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:09.121892 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:09.122183 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:09.122184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:09.122186 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:10:09.122253 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:09.122254 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.123202 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.123203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:09.124376 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:09.124378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.125205 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.125206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:09.125294 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:09.125308 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 07:10:09.125408 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 07:10:09.125460 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":84,"threads_started":1}
I20260924 07:10:09.125551 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:09.121703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39354 (local address 127.0.98.3:44545)
0924 07:10:09.122028 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:09.122033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:09.122043 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:10:09.122084 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:09.122086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:09.122101 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:09.122186 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:10:09.122334 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.123122 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.124472 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.125105 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.125202 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:09.125279 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:09.125426 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 07:10:09.125464 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":99,"threads_started":1}
I20260924 07:10:09.125913 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:09.121526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42777 (local address 127.0.0.1:47584)
0924 07:10:09.121677 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:09.121718 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:09.121784 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:09.122101 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:09.122104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:09.122106 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:10:09.122171 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:09.122173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.123601 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.123602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:09.124661 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:09.124663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.125667 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.125668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:09.125730 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:09.125734 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:10:09.125798 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 07:10:09.125838 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":43,"threads_started":1}
I20260924 07:10:09.126044 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:09.121626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47584 (local address 127.0.98.2:42777)
0924 07:10:09.121927 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:09.121931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:09.121940 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:10:09.121969 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:09.121971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:09.121983 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:09.122087 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:10:09.122249 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.123499 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.124767 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.125480 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.125555 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:09.125628 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:09.125827 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 07:10:09.125871 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":123,"threads_started":1}
I20260924 07:10:09.126111 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:09.121909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43628 (local address 127.0.98.1:45771)
0924 07:10:09.122078 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:09.122081 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:09.122352 (+ 271us) server_negotiation.cc:407] Connection header received
0924 07:10:09.122594 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:09.122596 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:09.122613 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:09.122689 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:10:09.122812 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.123754 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.124858 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.125519 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.125574 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:09.125640 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:09.125951 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 07:10:09.126002 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":78,"threads_started":1}
I20260924 07:10:09.126276 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:09.121702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45771 (local address 127.0.0.1:43628)
0924 07:10:09.122278 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:09.122305 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:09.122356 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:09.122705 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:09.122705 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:09.122707 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:10:09.122750 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:09.122751 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.123840 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.123841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:09.124773 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:09.124775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.125680 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.125681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:09.125826 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:09.125829 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:10:09.125867 (+ 38us) client_negotiation.cc:240] Negotiation successful
0924 07:10:09.125898 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":68,"threads_started":1}
I20260924 07:10:09.126396 8322 tablet_service.cc:1467] Processing CreateTablet for tablet ecd9826df002440fa29677ea7710c114 (DEFAULT_TABLE table=test-table [id=628229dccb744247923e05144d2f2d60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:10:09.126802 8322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecd9826df002440fa29677ea7710c114. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:09.126824 8042 tablet_service.cc:1467] Processing CreateTablet for tablet ecd9826df002440fa29677ea7710c114 (DEFAULT_TABLE table=test-table [id=628229dccb744247923e05144d2f2d60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:10:09.127063 8042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecd9826df002440fa29677ea7710c114. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:09.129935 8416 tablet_bootstrap.cc:492] T ecd9826df002440fa29677ea7710c114 P 3ba1e1457a494fc58f23c23c7e156a59: Bootstrap starting.
I20260924 07:10:09.130352 8417 tablet_bootstrap.cc:492] T ecd9826df002440fa29677ea7710c114 P 83906483277942bab937e411fc4c666e: Bootstrap starting.
I20260924 07:10:09.130280 8182 tablet_service.cc:1467] Processing CreateTablet for tablet ecd9826df002440fa29677ea7710c114 (DEFAULT_TABLE table=test-table [id=628229dccb744247923e05144d2f2d60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:10:09.130600 8182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecd9826df002440fa29677ea7710c114. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:09.130697 8416 tablet_bootstrap.cc:654] T ecd9826df002440fa29677ea7710c114 P 3ba1e1457a494fc58f23c23c7e156a59: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:09.130964 8417 tablet_bootstrap.cc:654] T ecd9826df002440fa29677ea7710c114 P 83906483277942bab937e411fc4c666e: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:09.130975 8416 log.cc:824] T ecd9826df002440fa29677ea7710c114 P 3ba1e1457a494fc58f23c23c7e156a59: Log is configured to *not* fsync() on all Append() calls
I20260924 07:10:09.131299 8417 log.cc:824] T ecd9826df002440fa29677ea7710c114 P 83906483277942bab937e411fc4c666e: Log is configured to *not* fsync() on all Append() calls
I20260924 07:10:09.131726 8416 tablet_bootstrap.cc:492] T ecd9826df002440fa29677ea7710c114 P 3ba1e1457a494fc58f23c23c7e156a59: No bootstrap required, opened a new log
I20260924 07:10:09.131812 8416 ts_tablet_manager.cc:1397] T ecd9826df002440fa29677ea7710c114 P 3ba1e1457a494fc58f23c23c7e156a59: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:10:09.132030 8417 tablet_bootstrap.cc:492] T ecd9826df002440fa29677ea7710c114 P 83906483277942bab937e411fc4c666e: No bootstrap required, opened a new log
I20260924 07:10:09.132099 8417 ts_tablet_manager.cc:1397] T ecd9826df002440fa29677ea7710c114 P 83906483277942bab937e411fc4c666e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:10:09.133869 8420 tablet_bootstrap.cc:492] T ecd9826df002440fa29677ea7710c114 P 5f1865a617384ee3aa88bc78243d7949: Bootstrap starting.
I20260924 07:10:09.133844 8416 raft_consensus.cc:348] T ecd9826df002440fa29677ea7710c114 P 3ba1e1457a494fc58f23c23c7e156a59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.134017 8416 raft_consensus.cc:374] T ecd9826df002440fa29677ea7710c114 P 3ba1e1457a494fc58f23c23c7e156a59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:09.134063 8416 raft_consensus.cc:729] T ecd9826df002440fa29677ea7710c114 P 3ba1e1457a494fc58f23c23c7e156a59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ba1e1457a494fc58f23c23c7e156a59, State: Initialized, Role: FOLLOWER
I20260924 07:10:09.134004 8417 raft_consensus.cc:348] T ecd9826df002440fa29677ea7710c114 P 83906483277942bab937e411fc4c666e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.134150 8417 raft_consensus.cc:374] T ecd9826df002440fa29677ea7710c114 P 83906483277942bab937e411fc4c666e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:09.134196 8417 raft_consensus.cc:729] T ecd9826df002440fa29677ea7710c114 P 83906483277942bab937e411fc4c666e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83906483277942bab937e411fc4c666e, State: Initialized, Role: FOLLOWER
I20260924 07:10:09.134230 8416 consensus_queue.cc:260] T ecd9826df002440fa29677ea7710c114 P 3ba1e1457a494fc58f23c23c7e156a59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.134320 8417 consensus_queue.cc:260] T ecd9826df002440fa29677ea7710c114 P 83906483277942bab937e411fc4c666e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.134502 8416 ts_tablet_manager.cc:1428] T ecd9826df002440fa29677ea7710c114 P 3ba1e1457a494fc58f23c23c7e156a59: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260924 07:10:09.134552 8417 ts_tablet_manager.cc:1428] T ecd9826df002440fa29677ea7710c114 P 83906483277942bab937e411fc4c666e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 07:10:09.134624 8388 heartbeater.cc:503] Master 127.0.98.62:34025 was elected leader, sending a full tablet report...
I20260924 07:10:09.134663 8108 heartbeater.cc:503] Master 127.0.98.62:34025 was elected leader, sending a full tablet report...
I20260924 07:10:09.134655 8420 tablet_bootstrap.cc:654] T ecd9826df002440fa29677ea7710c114 P 5f1865a617384ee3aa88bc78243d7949: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:09.135001 8420 log.cc:824] T ecd9826df002440fa29677ea7710c114 P 5f1865a617384ee3aa88bc78243d7949: Log is configured to *not* fsync() on all Append() calls
I20260924 07:10:09.135943 8420 tablet_bootstrap.cc:492] T ecd9826df002440fa29677ea7710c114 P 5f1865a617384ee3aa88bc78243d7949: No bootstrap required, opened a new log
I20260924 07:10:09.135993 8420 ts_tablet_manager.cc:1397] T ecd9826df002440fa29677ea7710c114 P 5f1865a617384ee3aa88bc78243d7949: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:10:09.137430 8420 raft_consensus.cc:348] T ecd9826df002440fa29677ea7710c114 P 5f1865a617384ee3aa88bc78243d7949 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.137548 8420 raft_consensus.cc:374] T ecd9826df002440fa29677ea7710c114 P 5f1865a617384ee3aa88bc78243d7949 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:09.137569 8420 raft_consensus.cc:729] T ecd9826df002440fa29677ea7710c114 P 5f1865a617384ee3aa88bc78243d7949 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f1865a617384ee3aa88bc78243d7949, State: Initialized, Role: FOLLOWER
I20260924 07:10:09.137701 8420 consensus_queue.cc:260] T ecd9826df002440fa29677ea7710c114 P 5f1865a617384ee3aa88bc78243d7949 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.137943 8420 ts_tablet_manager.cc:1428] T ecd9826df002440fa29677ea7710c114 P 5f1865a617384ee3aa88bc78243d7949: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:10:09.137970 8248 heartbeater.cc:503] Master 127.0.98.62:34025 was elected leader, sending a full tablet report...
W20260924 07:10:09.164427 8249 tablet.cc:2369] T ecd9826df002440fa29677ea7710c114 P 5f1865a617384ee3aa88bc78243d7949: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:10:09.249795 8421 raft_consensus.cc:482] T ecd9826df002440fa29677ea7710c114 P 3ba1e1457a494fc58f23c23c7e156a59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:10:09.249980 8421 raft_consensus.cc:504] T ecd9826df002440fa29677ea7710c114 P 3ba1e1457a494fc58f23c23c7e156a59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.250360 8421 leader_election.cc:288] T ecd9826df002440fa29677ea7710c114 P 3ba1e1457a494fc58f23c23c7e156a59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f1865a617384ee3aa88bc78243d7949 (127.0.98.2:42777), 83906483277942bab937e411fc4c666e (127.0.98.1:45771)
I20260924 07:10:09.253870 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:09.250624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59545 (local address 127.0.98.2:42777)
0924 07:10:09.250686 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:09.250689 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:09.250738 (+ 49us) server_negotiation.cc:407] Connection header received
0924 07:10:09.250871 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:09.250872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:09.250882 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:09.250940 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:10:09.251102 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.251657 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.253131 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.253493 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.253553 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:09.253562 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:09.253758 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 07:10:09.253803 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 07:10:09.253865 8281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:09.250564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42777 (local address 127.0.98.3:59545)
0924 07:10:09.250632 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:09.250664 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:09.250747 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:09.250959 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:09.250960 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:09.250961 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:10:09.251030 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:09.251032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.251740 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.251759 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:09.253048 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:09.253050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.253578 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.253579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:09.253679 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:09.253683 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:10:09.253733 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 07:10:09.253766 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":39}
I20260924 07:10:09.253962 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:09.250771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37035 (local address 127.0.98.1:45771)
0924 07:10:09.250813 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:09.250815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:09.250939 (+ 124us) server_negotiation.cc:407] Connection header received
0924 07:10:09.251009 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:09.251010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:09.251022 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:09.251094 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:10:09.251249 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.251731 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.253168 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.253518 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.253610 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:09.253621 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:09.253841 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 07:10:09.253885 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 07:10:09.254007 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:09.250669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45771 (local address 127.0.98.3:37035)
0924 07:10:09.250870 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:09.250879 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:09.250952 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:09.251096 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:09.251096 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:09.251098 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:10:09.251173 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:09.251174 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.251818 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.251819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:09.253049 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:09.253050 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.253668 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.253669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:09.253752 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:09.253757 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:10:09.253815 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 07:10:09.253856 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":5,"thread_start_us":133,"threads_started":1}
I20260924 07:10:09.254226 8202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecd9826df002440fa29677ea7710c114" candidate_uuid: "3ba1e1457a494fc58f23c23c7e156a59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1865a617384ee3aa88bc78243d7949" is_pre_election: true
I20260924 07:10:09.254372 8202 raft_consensus.cc:2454] T ecd9826df002440fa29677ea7710c114 P 5f1865a617384ee3aa88bc78243d7949 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ba1e1457a494fc58f23c23c7e156a59 in term 0.
I20260924 07:10:09.254400 8062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecd9826df002440fa29677ea7710c114" candidate_uuid: "3ba1e1457a494fc58f23c23c7e156a59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83906483277942bab937e411fc4c666e" is_pre_election: true
I20260924 07:10:09.254572 8062 raft_consensus.cc:2454] T ecd9826df002440fa29677ea7710c114 P 83906483277942bab937e411fc4c666e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ba1e1457a494fc58f23c23c7e156a59 in term 0.
I20260924 07:10:09.254578 8274 leader_election.cc:302] T ecd9826df002440fa29677ea7710c114 P 3ba1e1457a494fc58f23c23c7e156a59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ba1e1457a494fc58f23c23c7e156a59, 5f1865a617384ee3aa88bc78243d7949; no voters:
I20260924 07:10:09.254714 8421 raft_consensus.cc:2781] T ecd9826df002440fa29677ea7710c114 P 3ba1e1457a494fc58f23c23c7e156a59 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260924 07:10:09.254750 8109 tablet.cc:2369] T ecd9826df002440fa29677ea7710c114 P 83906483277942bab937e411fc4c666e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:10:09.254782 8421 raft_consensus.cc:482] T ecd9826df002440fa29677ea7710c114 P 3ba1e1457a494fc58f23c23c7e156a59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:10:09.254817 8421 raft_consensus.cc:3037] T ecd9826df002440fa29677ea7710c114 P 3ba1e1457a494fc58f23c23c7e156a59 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:09.255882 8421 raft_consensus.cc:504] T ecd9826df002440fa29677ea7710c114 P 3ba1e1457a494fc58f23c23c7e156a59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.256014 8421 leader_election.cc:288] T ecd9826df002440fa29677ea7710c114 P 3ba1e1457a494fc58f23c23c7e156a59 [CANDIDATE]: Term 1 election: Requested vote from peers 5f1865a617384ee3aa88bc78243d7949 (127.0.98.2:42777), 83906483277942bab937e411fc4c666e (127.0.98.1:45771)
I20260924 07:10:09.256193 8202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecd9826df002440fa29677ea7710c114" candidate_uuid: "3ba1e1457a494fc58f23c23c7e156a59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1865a617384ee3aa88bc78243d7949"
I20260924 07:10:09.256259 8062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecd9826df002440fa29677ea7710c114" candidate_uuid: "3ba1e1457a494fc58f23c23c7e156a59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83906483277942bab937e411fc4c666e"
I20260924 07:10:09.256290 8202 raft_consensus.cc:3037] T ecd9826df002440fa29677ea7710c114 P 5f1865a617384ee3aa88bc78243d7949 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:09.256353 8062 raft_consensus.cc:3037] T ecd9826df002440fa29677ea7710c114 P 83906483277942bab937e411fc4c666e [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:09.257085 8062 raft_consensus.cc:2454] T ecd9826df002440fa29677ea7710c114 P 83906483277942bab937e411fc4c666e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba1e1457a494fc58f23c23c7e156a59 in term 1.
I20260924 07:10:09.257085 8202 raft_consensus.cc:2454] T ecd9826df002440fa29677ea7710c114 P 5f1865a617384ee3aa88bc78243d7949 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba1e1457a494fc58f23c23c7e156a59 in term 1.
I20260924 07:10:09.257328 8274 leader_election.cc:302] T ecd9826df002440fa29677ea7710c114 P 3ba1e1457a494fc58f23c23c7e156a59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ba1e1457a494fc58f23c23c7e156a59, 83906483277942bab937e411fc4c666e; no voters:
I20260924 07:10:09.257467 8421 raft_consensus.cc:2781] T ecd9826df002440fa29677ea7710c114 P 3ba1e1457a494fc58f23c23c7e156a59 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:10:09.257645 8421 raft_consensus.cc:686] T ecd9826df002440fa29677ea7710c114 P 3ba1e1457a494fc58f23c23c7e156a59 [term 1 LEADER]: Becoming Leader. State: Replica: 3ba1e1457a494fc58f23c23c7e156a59, State: Running, Role: LEADER
I20260924 07:10:09.257745 8421 consensus_queue.cc:237] T ecd9826df002440fa29677ea7710c114 P 3ba1e1457a494fc58f23c23c7e156a59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.258627 7905 catalog_manager.cc:5495] T ecd9826df002440fa29677ea7710c114 P 3ba1e1457a494fc58f23c23c7e156a59 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ba1e1457a494fc58f23c23c7e156a59 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3ba1e1457a494fc58f23c23c7e156a59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } health_report { overall_health: HEALTHY } } }
I20260924 07:10:09.272521 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:09.270112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39370 (local address 127.0.98.3:44545)
0924 07:10:09.270165 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:09.270167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:09.270177 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:10:09.270268 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:09.270273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:09.270290 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:09.270366 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:10:09.270496 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.271006 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.271773 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.271959 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.272034 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:09.272218 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:10:09.272304 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:10:09.272388 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:09.272429 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 07:10:09.272446 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 07:10:09.274245 8202 raft_consensus.cc:1261] T ecd9826df002440fa29677ea7710c114 P 5f1865a617384ee3aa88bc78243d7949 [term 1 FOLLOWER]: Refusing update from remote peer 3ba1e1457a494fc58f23c23c7e156a59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:10:09.274251 8062 raft_consensus.cc:1261] T ecd9826df002440fa29677ea7710c114 P 83906483277942bab937e411fc4c666e [term 1 FOLLOWER]: Refusing update from remote peer 3ba1e1457a494fc58f23c23c7e156a59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:10:09.274566 8421 consensus_queue.cc:1035] T ecd9826df002440fa29677ea7710c114 P 3ba1e1457a494fc58f23c23c7e156a59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:10:09.274722 8426 consensus_queue.cc:1035] T ecd9826df002440fa29677ea7710c114 P 3ba1e1457a494fc58f23c23c7e156a59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:10:09.277629 8432 mvcc.cc:205] Tried to move back new op lower bound from 7332797682784468992 to 7332797682720354304. Current Snapshot: MvccSnapshot[applied={T|T < 7332797682784468992}]
I20260924 07:10:09.278894 7904 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:10:09.278987 7904 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:10:09.279675 7904 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32fa0114c29249baacf9a200be709192: Sending DeleteTablet for 3 replicas of tablet ecd9826df002440fa29677ea7710c114
I20260924 07:10:09.279981 8322 tablet_service.cc:1514] Processing DeleteTablet for tablet ecd9826df002440fa29677ea7710c114 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:10:09 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39354
I20260924 07:10:09.280151 8182 tablet_service.cc:1514] Processing DeleteTablet for tablet ecd9826df002440fa29677ea7710c114 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:10:09 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47584
I20260924 07:10:09.280252 8042 tablet_service.cc:1514] Processing DeleteTablet for tablet ecd9826df002440fa29677ea7710c114 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:10:09 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43628
I20260924 07:10:09.280300 8438 tablet_replica.cc:330] T ecd9826df002440fa29677ea7710c114 P 3ba1e1457a494fc58f23c23c7e156a59: stopping tablet replica
I20260924 07:10:09.280386 8438 raft_consensus.cc:2229] T ecd9826df002440fa29677ea7710c114 P 3ba1e1457a494fc58f23c23c7e156a59 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:10:09.280474 8438 raft_consensus.cc:2258] T ecd9826df002440fa29677ea7710c114 P 3ba1e1457a494fc58f23c23c7e156a59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:10:09.280638 8440 tablet_replica.cc:330] T ecd9826df002440fa29677ea7710c114 P 83906483277942bab937e411fc4c666e: stopping tablet replica
I20260924 07:10:09.280623 7904 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 07:10:09.280725 8440 raft_consensus.cc:2229] T ecd9826df002440fa29677ea7710c114 P 83906483277942bab937e411fc4c666e [term 1 FOLLOWER]: Raft consensus shutting down.
W20260924 07:10:09.280777 7904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:10:09.280843 8440 raft_consensus.cc:2258] T ecd9826df002440fa29677ea7710c114 P 83906483277942bab937e411fc4c666e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:10:09.281270 8440 ts_tablet_manager.cc:1905] T ecd9826df002440fa29677ea7710c114 P 83906483277942bab937e411fc4c666e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:10:09.281853 8042 tablet_service.cc:1467] Processing CreateTablet for tablet f388e29619ba4477b5c48013269e059e (DEFAULT_TABLE table=test-table [id=9c27dfa2be84473e807bc42a3db794e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:10:09.281961 8042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f388e29619ba4477b5c48013269e059e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:09.281939 8322 tablet_service.cc:1467] Processing CreateTablet for tablet f388e29619ba4477b5c48013269e059e (DEFAULT_TABLE table=test-table [id=9c27dfa2be84473e807bc42a3db794e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:10:09.282040 8322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f388e29619ba4477b5c48013269e059e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:09.282829 8439 tablet_replica.cc:330] T ecd9826df002440fa29677ea7710c114 P 5f1865a617384ee3aa88bc78243d7949: stopping tablet replica
I20260924 07:10:09.282913 8439 raft_consensus.cc:2229] T ecd9826df002440fa29677ea7710c114 P 5f1865a617384ee3aa88bc78243d7949 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:10:09.282995 8439 raft_consensus.cc:2258] T ecd9826df002440fa29677ea7710c114 P 5f1865a617384ee3aa88bc78243d7949 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:10:09.283243 8417 tablet_bootstrap.cc:492] T f388e29619ba4477b5c48013269e059e P 83906483277942bab937e411fc4c666e: Bootstrap starting.
I20260924 07:10:09.283401 8416 tablet_bootstrap.cc:492] T f388e29619ba4477b5c48013269e059e P 3ba1e1457a494fc58f23c23c7e156a59: Bootstrap starting.
I20260924 07:10:09.283576 8440 ts_tablet_manager.cc:1918] T ecd9826df002440fa29677ea7710c114 P 83906483277942bab937e411fc4c666e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:10:09.283756 8440 log.cc:1196] T ecd9826df002440fa29677ea7710c114 P 83906483277942bab937e411fc4c666e: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-0/wal/wals/ecd9826df002440fa29677ea7710c114
I20260924 07:10:09.283820 8416 tablet_bootstrap.cc:654] T f388e29619ba4477b5c48013269e059e P 3ba1e1457a494fc58f23c23c7e156a59: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:09.283622 8417 tablet_bootstrap.cc:654] T f388e29619ba4477b5c48013269e059e P 83906483277942bab937e411fc4c666e: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:09.284197 8440 ts_tablet_manager.cc:1939] T ecd9826df002440fa29677ea7710c114 P 83906483277942bab937e411fc4c666e: Deleting consensus metadata
I20260924 07:10:09.284302 8439 ts_tablet_manager.cc:1905] T ecd9826df002440fa29677ea7710c114 P 5f1865a617384ee3aa88bc78243d7949: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:10:09.284384 8182 tablet_service.cc:1467] Processing CreateTablet for tablet f388e29619ba4477b5c48013269e059e (DEFAULT_TABLE table=test-table [id=9c27dfa2be84473e807bc42a3db794e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:10:09.284536 8182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f388e29619ba4477b5c48013269e059e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:09.284809 7890 catalog_manager.cc:4841] TS 83906483277942bab937e411fc4c666e (127.0.98.1:45771): tablet ecd9826df002440fa29677ea7710c114 (table test-table [id=628229dccb744247923e05144d2f2d60]) successfully deleted
I20260924 07:10:09.285969 8439 ts_tablet_manager.cc:1918] T ecd9826df002440fa29677ea7710c114 P 5f1865a617384ee3aa88bc78243d7949: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:10:09.286070 8417 tablet_bootstrap.cc:492] T f388e29619ba4477b5c48013269e059e P 83906483277942bab937e411fc4c666e: No bootstrap required, opened a new log
I20260924 07:10:09.286017 8420 tablet_bootstrap.cc:492] T f388e29619ba4477b5c48013269e059e P 5f1865a617384ee3aa88bc78243d7949: Bootstrap starting.
I20260924 07:10:09.286135 8417 ts_tablet_manager.cc:1397] T f388e29619ba4477b5c48013269e059e P 83906483277942bab937e411fc4c666e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 07:10:09.286100 8439 log.cc:1196] T ecd9826df002440fa29677ea7710c114 P 5f1865a617384ee3aa88bc78243d7949: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-1/wal/wals/ecd9826df002440fa29677ea7710c114
I20260924 07:10:09.286478 8420 tablet_bootstrap.cc:654] T f388e29619ba4477b5c48013269e059e P 5f1865a617384ee3aa88bc78243d7949: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:09.286468 8417 raft_consensus.cc:348] T f388e29619ba4477b5c48013269e059e P 83906483277942bab937e411fc4c666e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.286546 8417 raft_consensus.cc:374] T f388e29619ba4477b5c48013269e059e P 83906483277942bab937e411fc4c666e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:09.286572 8417 raft_consensus.cc:729] T f388e29619ba4477b5c48013269e059e P 83906483277942bab937e411fc4c666e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83906483277942bab937e411fc4c666e, State: Initialized, Role: FOLLOWER
I20260924 07:10:09.286615 8417 consensus_queue.cc:260] T f388e29619ba4477b5c48013269e059e P 83906483277942bab937e411fc4c666e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.286641 8439 ts_tablet_manager.cc:1939] T ecd9826df002440fa29677ea7710c114 P 5f1865a617384ee3aa88bc78243d7949: Deleting consensus metadata
I20260924 07:10:09.286738 8417 ts_tablet_manager.cc:1428] T f388e29619ba4477b5c48013269e059e P 83906483277942bab937e411fc4c666e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:10:09.287063 8438 ts_tablet_manager.cc:1905] T ecd9826df002440fa29677ea7710c114 P 3ba1e1457a494fc58f23c23c7e156a59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:10:09.287124 7890 catalog_manager.cc:4841] TS 5f1865a617384ee3aa88bc78243d7949 (127.0.98.2:42777): tablet ecd9826df002440fa29677ea7710c114 (table test-table [id=628229dccb744247923e05144d2f2d60]) successfully deleted
I20260924 07:10:09.288614 8438 ts_tablet_manager.cc:1918] T ecd9826df002440fa29677ea7710c114 P 3ba1e1457a494fc58f23c23c7e156a59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:10:09.288699 8438 log.cc:1196] T ecd9826df002440fa29677ea7710c114 P 3ba1e1457a494fc58f23c23c7e156a59: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-2/wal/wals/ecd9826df002440fa29677ea7710c114
I20260924 07:10:09.288820 8416 tablet_bootstrap.cc:492] T f388e29619ba4477b5c48013269e059e P 3ba1e1457a494fc58f23c23c7e156a59: No bootstrap required, opened a new log
I20260924 07:10:09.288872 8416 ts_tablet_manager.cc:1397] T f388e29619ba4477b5c48013269e059e P 3ba1e1457a494fc58f23c23c7e156a59: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260924 07:10:09.289021 8438 ts_tablet_manager.cc:1939] T ecd9826df002440fa29677ea7710c114 P 3ba1e1457a494fc58f23c23c7e156a59: Deleting consensus metadata
I20260924 07:10:09.288998 8416 raft_consensus.cc:348] T f388e29619ba4477b5c48013269e059e P 3ba1e1457a494fc58f23c23c7e156a59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.289074 8416 raft_consensus.cc:374] T f388e29619ba4477b5c48013269e059e P 3ba1e1457a494fc58f23c23c7e156a59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:09.289110 8416 raft_consensus.cc:729] T f388e29619ba4477b5c48013269e059e P 3ba1e1457a494fc58f23c23c7e156a59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ba1e1457a494fc58f23c23c7e156a59, State: Initialized, Role: FOLLOWER
I20260924 07:10:09.289167 8416 consensus_queue.cc:260] T f388e29619ba4477b5c48013269e059e P 3ba1e1457a494fc58f23c23c7e156a59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.289284 8416 ts_tablet_manager.cc:1428] T f388e29619ba4477b5c48013269e059e P 3ba1e1457a494fc58f23c23c7e156a59: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:09.289386 8420 tablet_bootstrap.cc:492] T f388e29619ba4477b5c48013269e059e P 5f1865a617384ee3aa88bc78243d7949: No bootstrap required, opened a new log
I20260924 07:10:09.289426 8420 ts_tablet_manager.cc:1397] T f388e29619ba4477b5c48013269e059e P 5f1865a617384ee3aa88bc78243d7949: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 07:10:09.289563 8420 raft_consensus.cc:348] T f388e29619ba4477b5c48013269e059e P 5f1865a617384ee3aa88bc78243d7949 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.289638 8420 raft_consensus.cc:374] T f388e29619ba4477b5c48013269e059e P 5f1865a617384ee3aa88bc78243d7949 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:09.289659 8420 raft_consensus.cc:729] T f388e29619ba4477b5c48013269e059e P 5f1865a617384ee3aa88bc78243d7949 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f1865a617384ee3aa88bc78243d7949, State: Initialized, Role: FOLLOWER
I20260924 07:10:09.289723 8420 consensus_queue.cc:260] T f388e29619ba4477b5c48013269e059e P 5f1865a617384ee3aa88bc78243d7949 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.289762 7889 catalog_manager.cc:4841] TS 3ba1e1457a494fc58f23c23c7e156a59 (127.0.98.3:44545): tablet ecd9826df002440fa29677ea7710c114 (table test-table [id=628229dccb744247923e05144d2f2d60]) successfully deleted
I20260924 07:10:09.289813 8420 ts_tablet_manager.cc:1428] T f388e29619ba4477b5c48013269e059e P 5f1865a617384ee3aa88bc78243d7949: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260924 07:10:09.331838 8389 tablet.cc:2369] T f388e29619ba4477b5c48013269e059e P 3ba1e1457a494fc58f23c23c7e156a59: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:10:09.451437 8422 raft_consensus.cc:482] T f388e29619ba4477b5c48013269e059e P 83906483277942bab937e411fc4c666e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:10:09.451615 8422 raft_consensus.cc:504] T f388e29619ba4477b5c48013269e059e P 83906483277942bab937e411fc4c666e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.452016 8422 leader_election.cc:288] T f388e29619ba4477b5c48013269e059e P 83906483277942bab937e411fc4c666e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f1865a617384ee3aa88bc78243d7949 (127.0.98.2:42777), 3ba1e1457a494fc58f23c23c7e156a59 (127.0.98.3:44545)
I20260924 07:10:09.455155 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:09.452238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44545 (local address 127.0.98.1:53313)
0924 07:10:09.452447 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:09.452456 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:09.452578 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:09.452706 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:09.452707 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:09.452710 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:10:09.452786 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:09.452787 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.453491 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.453492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:09.454286 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:09.454288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.454888 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.454890 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:09.454963 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:09.454966 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:10:09.455023 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 07:10:09.455060 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":50,"thread_start_us":102,"threads_started":1}
I20260924 07:10:09.455163 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:09.452295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53313 (local address 127.0.98.3:44545)
0924 07:10:09.452351 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:09.452353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:09.452560 (+ 207us) server_negotiation.cc:407] Connection header received
0924 07:10:09.452638 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:09.452639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:09.452648 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:09.452705 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:10:09.452880 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.453376 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.454386 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.454783 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.454869 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:09.454882 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:09.455073 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 07:10:09.455112 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 07:10:09.455574 8424 raft_consensus.cc:482] T f388e29619ba4477b5c48013269e059e P 5f1865a617384ee3aa88bc78243d7949 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:10:09.455643 8424 raft_consensus.cc:504] T f388e29619ba4477b5c48013269e059e P 5f1865a617384ee3aa88bc78243d7949 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.455615 8337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f388e29619ba4477b5c48013269e059e" candidate_uuid: "83906483277942bab937e411fc4c666e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ba1e1457a494fc58f23c23c7e156a59" is_pre_election: true
I20260924 07:10:09.455749 8337 raft_consensus.cc:2454] T f388e29619ba4477b5c48013269e059e P 3ba1e1457a494fc58f23c23c7e156a59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83906483277942bab937e411fc4c666e in term 0.
I20260924 07:10:09.455931 8424 leader_election.cc:288] T f388e29619ba4477b5c48013269e059e P 5f1865a617384ee3aa88bc78243d7949 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83906483277942bab937e411fc4c666e (127.0.98.1:45771), 3ba1e1457a494fc58f23c23c7e156a59 (127.0.98.3:44545)
I20260924 07:10:09.456017 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:09.452319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37035 (local address 127.0.98.2:42777)
0924 07:10:09.452396 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:09.452398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:09.452509 (+ 111us) server_negotiation.cc:407] Connection header received
0924 07:10:09.452549 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:09.452550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:09.452561 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:09.452615 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:10:09.452849 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.453334 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.454919 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.455464 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.455664 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:09.455674 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:09.455883 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 07:10:09.455921 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 07:10:09.455957 7993 leader_election.cc:302] T f388e29619ba4477b5c48013269e059e P 83906483277942bab937e411fc4c666e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ba1e1457a494fc58f23c23c7e156a59, 83906483277942bab937e411fc4c666e; no voters:
I20260924 07:10:09.456120 8422 raft_consensus.cc:2781] T f388e29619ba4477b5c48013269e059e P 83906483277942bab937e411fc4c666e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:10:09.456182 8422 raft_consensus.cc:482] T f388e29619ba4477b5c48013269e059e P 83906483277942bab937e411fc4c666e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:10:09.456224 8422 raft_consensus.cc:3037] T f388e29619ba4477b5c48013269e059e P 83906483277942bab937e411fc4c666e [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:09.456594 8445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:09.452237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42777 (local address 127.0.98.1:37035)
0924 07:10:09.452416 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:09.452427 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:09.452496 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:09.452630 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:09.452631 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:09.452633 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:10:09.452766 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:09.452769 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.453452 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.453453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:09.454282 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:09.454284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.455714 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.455715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:09.455791 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:09.455795 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:10:09.455950 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 07:10:09.456111 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":77,"threads_started":1}
I20260924 07:10:09.456809 8202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f388e29619ba4477b5c48013269e059e" candidate_uuid: "83906483277942bab937e411fc4c666e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1865a617384ee3aa88bc78243d7949" is_pre_election: true
I20260924 07:10:09.456907 8202 raft_consensus.cc:2454] T f388e29619ba4477b5c48013269e059e P 5f1865a617384ee3aa88bc78243d7949 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83906483277942bab937e411fc4c666e in term 0.
I20260924 07:10:09.457238 8422 raft_consensus.cc:504] T f388e29619ba4477b5c48013269e059e P 83906483277942bab937e411fc4c666e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.457374 8422 leader_election.cc:288] T f388e29619ba4477b5c48013269e059e P 83906483277942bab937e411fc4c666e [CANDIDATE]: Term 1 election: Requested vote from peers 5f1865a617384ee3aa88bc78243d7949 (127.0.98.2:42777), 3ba1e1457a494fc58f23c23c7e156a59 (127.0.98.3:44545)
I20260924 07:10:09.457553 8202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f388e29619ba4477b5c48013269e059e" candidate_uuid: "83906483277942bab937e411fc4c666e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1865a617384ee3aa88bc78243d7949"
I20260924 07:10:09.457633 8202 raft_consensus.cc:3037] T f388e29619ba4477b5c48013269e059e P 5f1865a617384ee3aa88bc78243d7949 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:09.457645 8337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f388e29619ba4477b5c48013269e059e" candidate_uuid: "83906483277942bab937e411fc4c666e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ba1e1457a494fc58f23c23c7e156a59"
I20260924 07:10:09.457742 8337 raft_consensus.cc:3037] T f388e29619ba4477b5c48013269e059e P 3ba1e1457a494fc58f23c23c7e156a59 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:09.458380 8202 raft_consensus.cc:2454] T f388e29619ba4477b5c48013269e059e P 5f1865a617384ee3aa88bc78243d7949 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83906483277942bab937e411fc4c666e in term 1.
I20260924 07:10:09.458729 8337 raft_consensus.cc:2454] T f388e29619ba4477b5c48013269e059e P 3ba1e1457a494fc58f23c23c7e156a59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83906483277942bab937e411fc4c666e in term 1.
I20260924 07:10:09.458935 7994 leader_election.cc:302] T f388e29619ba4477b5c48013269e059e P 83906483277942bab937e411fc4c666e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f1865a617384ee3aa88bc78243d7949, 83906483277942bab937e411fc4c666e; no voters:
I20260924 07:10:09.459020 8422 raft_consensus.cc:2781] T f388e29619ba4477b5c48013269e059e P 83906483277942bab937e411fc4c666e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:10:09.459156 8422 raft_consensus.cc:686] T f388e29619ba4477b5c48013269e059e P 83906483277942bab937e411fc4c666e [term 1 LEADER]: Becoming Leader. State: Replica: 83906483277942bab937e411fc4c666e, State: Running, Role: LEADER
I20260924 07:10:09.459224 8422 consensus_queue.cc:237] T f388e29619ba4477b5c48013269e059e P 83906483277942bab937e411fc4c666e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.459901 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:09.456236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37733 (local address 127.0.98.3:44545)
0924 07:10:09.456393 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:09.456395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:09.456480 (+ 85us) server_negotiation.cc:407] Connection header received
0924 07:10:09.456573 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:09.456574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:09.456585 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:09.456615 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:10:09.456838 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.457408 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.458969 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.459517 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.459584 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:09.459592 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:09.459790 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 07:10:09.459827 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260924 07:10:09.459833 7905 catalog_manager.cc:5495] T f388e29619ba4477b5c48013269e059e P 83906483277942bab937e411fc4c666e reported cstate change: term changed from 0 to 1, leader changed from <none> to 83906483277942bab937e411fc4c666e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "83906483277942bab937e411fc4c666e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } health_report { overall_health: UNKNOWN } } }
I20260924 07:10:09.460005 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:09.456324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45771 (local address 127.0.98.2:47079)
0924 07:10:09.456436 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:09.456446 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:09.456541 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:09.456880 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:09.456881 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:09.456883 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:10:09.457008 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:09.457010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.457846 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.457847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:09.459038 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:09.459041 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.459774 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.459775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:09.459835 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:09.459839 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:10:09.459904 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 07:10:09.459941 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":55,"threads_started":1}
I20260924 07:10:09.460026 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:09.456427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47079 (local address 127.0.98.1:45771)
0924 07:10:09.456466 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:09.456468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:09.456515 (+ 47us) server_negotiation.cc:407] Connection header received
0924 07:10:09.456608 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:09.456609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:09.456617 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:09.456644 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:10:09.457106 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.457745 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.459180 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.459698 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.459771 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:09.459779 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:09.459926 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 07:10:09.459964 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 07:10:09.460156 8447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:09.456113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44545 (local address 127.0.98.2:37733)
0924 07:10:09.456417 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:09.456428 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:09.456506 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:09.456677 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:09.456678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:09.456680 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:10:09.456760 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:09.456762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.457489 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.457489 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:09.458880 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:09.458882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.459625 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.459626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:09.459704 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:09.459708 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:10:09.460009 (+ 301us) client_negotiation.cc:240] Negotiation successful
0924 07:10:09.460057 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":52,"thread_start_us":54,"threads_started":1}
I20260924 07:10:09.460433 8337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f388e29619ba4477b5c48013269e059e" candidate_uuid: "5f1865a617384ee3aa88bc78243d7949" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ba1e1457a494fc58f23c23c7e156a59" is_pre_election: true
I20260924 07:10:09.460464 8062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f388e29619ba4477b5c48013269e059e" candidate_uuid: "5f1865a617384ee3aa88bc78243d7949" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83906483277942bab937e411fc4c666e" is_pre_election: true
I20260924 07:10:09.460525 8337 raft_consensus.cc:2379] T f388e29619ba4477b5c48013269e059e P 3ba1e1457a494fc58f23c23c7e156a59 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5f1865a617384ee3aa88bc78243d7949 in current term 1: Already voted for candidate 83906483277942bab937e411fc4c666e in this term.
I20260924 07:10:09.460762 8133 leader_election.cc:302] T f388e29619ba4477b5c48013269e059e P 5f1865a617384ee3aa88bc78243d7949 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5f1865a617384ee3aa88bc78243d7949; no voters: 3ba1e1457a494fc58f23c23c7e156a59, 83906483277942bab937e411fc4c666e
I20260924 07:10:09.460903 8424 raft_consensus.cc:2726] T f388e29619ba4477b5c48013269e059e P 5f1865a617384ee3aa88bc78243d7949 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Sep 24 07:10:09 dist-test-slave-14ww krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233808, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 07:10:09.495867 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:09.489355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33478 (local address 127.0.98.62:34025)
0924 07:10:09.489420 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:09.489422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:09.489537 (+ 115us) server_negotiation.cc:407] Connection header received
0924 07:10:09.489576 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:09.489577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:09.489585 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:09.489623 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:09.490112 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.490590 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.491245 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.491431 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.491482 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:09.493575 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:09.493576 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:09.493577 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:09.493584 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:09.494899 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:09.495232 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:09.495233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:09.495233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:09.495282 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:09.495448 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:09.495449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:09.495449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:09.495543 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:09.495591 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:09.495736 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 07:10:09.495804 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 07:10:09.496021 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:09.489301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34025 (local address 127.0.0.1:33478)
0924 07:10:09.489466 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:09.489476 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:09.489533 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:09.489639 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:09.489639 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:09.489940 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:09.489989 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:09.489991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.490662 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.490662 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:09.491173 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:09.491174 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.491496 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.491497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:09.491535 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:09.492075 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:09.492079 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:09.493479 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:09.494976 (+ 1497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:09.494981 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:09.494983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:09.495167 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:09.495342 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:09.495342 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:09.495343 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:09.495396 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:09.495595 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:09.495599 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:09.495636 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 07:10:09.495712 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 07:10:09.495881 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":59,"thread_start_us":100,"threads_started":1}
I20260924 07:10:09.498071 7905 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33478:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 07:10:09.498257 7905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:10:09.499732 8322 tablet_service.cc:1467] Processing CreateTablet for tablet 5d3a00f27fab4e0ba766328042632f12 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ffb9fc81c0ca4c44a6b1019425b72e23]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:10:09.499735 8182 tablet_service.cc:1467] Processing CreateTablet for tablet 5d3a00f27fab4e0ba766328042632f12 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ffb9fc81c0ca4c44a6b1019425b72e23]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:10:09.499797 8042 tablet_service.cc:1467] Processing CreateTablet for tablet 5d3a00f27fab4e0ba766328042632f12 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ffb9fc81c0ca4c44a6b1019425b72e23]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 07:10:09.499866 8322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d3a00f27fab4e0ba766328042632f12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:09.499874 8182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d3a00f27fab4e0ba766328042632f12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:09.499907 8042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d3a00f27fab4e0ba766328042632f12. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:09.501353 8416 tablet_bootstrap.cc:492] T 5d3a00f27fab4e0ba766328042632f12 P 3ba1e1457a494fc58f23c23c7e156a59: Bootstrap starting.
I20260924 07:10:09.501359 8417 tablet_bootstrap.cc:492] T 5d3a00f27fab4e0ba766328042632f12 P 83906483277942bab937e411fc4c666e: Bootstrap starting.
I20260924 07:10:09.501359 8420 tablet_bootstrap.cc:492] T 5d3a00f27fab4e0ba766328042632f12 P 5f1865a617384ee3aa88bc78243d7949: Bootstrap starting.
I20260924 07:10:09.501688 8416 tablet_bootstrap.cc:654] T 5d3a00f27fab4e0ba766328042632f12 P 3ba1e1457a494fc58f23c23c7e156a59: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:09.501788 8417 tablet_bootstrap.cc:654] T 5d3a00f27fab4e0ba766328042632f12 P 83906483277942bab937e411fc4c666e: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:09.501812 8420 tablet_bootstrap.cc:654] T 5d3a00f27fab4e0ba766328042632f12 P 5f1865a617384ee3aa88bc78243d7949: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:09.502285 8416 tablet_bootstrap.cc:492] T 5d3a00f27fab4e0ba766328042632f12 P 3ba1e1457a494fc58f23c23c7e156a59: No bootstrap required, opened a new log
I20260924 07:10:09.502349 8416 ts_tablet_manager.cc:1397] T 5d3a00f27fab4e0ba766328042632f12 P 3ba1e1457a494fc58f23c23c7e156a59: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 07:10:09.502455 8417 tablet_bootstrap.cc:492] T 5d3a00f27fab4e0ba766328042632f12 P 83906483277942bab937e411fc4c666e: No bootstrap required, opened a new log
I20260924 07:10:09.502504 8417 ts_tablet_manager.cc:1397] T 5d3a00f27fab4e0ba766328042632f12 P 83906483277942bab937e411fc4c666e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:10:09.502465 8416 raft_consensus.cc:348] T 5d3a00f27fab4e0ba766328042632f12 P 3ba1e1457a494fc58f23c23c7e156a59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.502535 8416 raft_consensus.cc:374] T 5d3a00f27fab4e0ba766328042632f12 P 3ba1e1457a494fc58f23c23c7e156a59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:09.502559 8416 raft_consensus.cc:729] T 5d3a00f27fab4e0ba766328042632f12 P 3ba1e1457a494fc58f23c23c7e156a59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ba1e1457a494fc58f23c23c7e156a59, State: Initialized, Role: FOLLOWER
I20260924 07:10:09.502615 8420 tablet_bootstrap.cc:492] T 5d3a00f27fab4e0ba766328042632f12 P 5f1865a617384ee3aa88bc78243d7949: No bootstrap required, opened a new log
I20260924 07:10:09.502606 8416 consensus_queue.cc:260] T 5d3a00f27fab4e0ba766328042632f12 P 3ba1e1457a494fc58f23c23c7e156a59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.502662 8420 ts_tablet_manager.cc:1397] T 5d3a00f27fab4e0ba766328042632f12 P 5f1865a617384ee3aa88bc78243d7949: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:10:09.502650 8417 raft_consensus.cc:348] T 5d3a00f27fab4e0ba766328042632f12 P 83906483277942bab937e411fc4c666e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.502703 8416 ts_tablet_manager.cc:1428] T 5d3a00f27fab4e0ba766328042632f12 P 3ba1e1457a494fc58f23c23c7e156a59: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:09.502728 8417 raft_consensus.cc:374] T 5d3a00f27fab4e0ba766328042632f12 P 83906483277942bab937e411fc4c666e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:09.502751 8417 raft_consensus.cc:729] T 5d3a00f27fab4e0ba766328042632f12 P 83906483277942bab937e411fc4c666e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83906483277942bab937e411fc4c666e, State: Initialized, Role: FOLLOWER
I20260924 07:10:09.502725 8421 tablet_replica.cc:439] T 5d3a00f27fab4e0ba766328042632f12 P 3ba1e1457a494fc58f23c23c7e156a59: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } } }
I20260924 07:10:09.502787 8417 consensus_queue.cc:260] T 5d3a00f27fab4e0ba766328042632f12 P 83906483277942bab937e411fc4c666e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.502810 8421 tablet_replica.cc:442] T 5d3a00f27fab4e0ba766328042632f12 P 3ba1e1457a494fc58f23c23c7e156a59: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:10:09.502803 8420 raft_consensus.cc:348] T 5d3a00f27fab4e0ba766328042632f12 P 5f1865a617384ee3aa88bc78243d7949 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.502872 8420 raft_consensus.cc:374] T 5d3a00f27fab4e0ba766328042632f12 P 5f1865a617384ee3aa88bc78243d7949 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:09.502882 8417 ts_tablet_manager.cc:1428] T 5d3a00f27fab4e0ba766328042632f12 P 83906483277942bab937e411fc4c666e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 07:10:09.502895 8420 raft_consensus.cc:729] T 5d3a00f27fab4e0ba766328042632f12 P 5f1865a617384ee3aa88bc78243d7949 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f1865a617384ee3aa88bc78243d7949, State: Initialized, Role: FOLLOWER
I20260924 07:10:09.502934 8420 consensus_queue.cc:260] T 5d3a00f27fab4e0ba766328042632f12 P 5f1865a617384ee3aa88bc78243d7949 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.502948 8422 tablet_replica.cc:439] T 5d3a00f27fab4e0ba766328042632f12 P 83906483277942bab937e411fc4c666e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } } }
I20260924 07:10:09.503018 8420 ts_tablet_manager.cc:1428] T 5d3a00f27fab4e0ba766328042632f12 P 5f1865a617384ee3aa88bc78243d7949: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:09.503022 8422 tablet_replica.cc:442] T 5d3a00f27fab4e0ba766328042632f12 P 83906483277942bab937e411fc4c666e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:10:09.503072 8424 tablet_replica.cc:439] T 5d3a00f27fab4e0ba766328042632f12 P 5f1865a617384ee3aa88bc78243d7949: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } } }
I20260924 07:10:09.503147 8424 tablet_replica.cc:442] T 5d3a00f27fab4e0ba766328042632f12 P 5f1865a617384ee3aa88bc78243d7949: This TxnStatusTablet replica's current role is: FOLLOWER
W20260924 07:10:09.535218 7946 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.0.98.62:34025
I20260924 07:10:09.560776 8421 raft_consensus.cc:482] T 5d3a00f27fab4e0ba766328042632f12 P 3ba1e1457a494fc58f23c23c7e156a59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:10:09.560916 8421 raft_consensus.cc:504] T 5d3a00f27fab4e0ba766328042632f12 P 3ba1e1457a494fc58f23c23c7e156a59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.561062 8421 leader_election.cc:288] T 5d3a00f27fab4e0ba766328042632f12 P 3ba1e1457a494fc58f23c23c7e156a59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83906483277942bab937e411fc4c666e (127.0.98.1:45771), 5f1865a617384ee3aa88bc78243d7949 (127.0.98.2:42777)
I20260924 07:10:09.561302 8062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3a00f27fab4e0ba766328042632f12" candidate_uuid: "3ba1e1457a494fc58f23c23c7e156a59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83906483277942bab937e411fc4c666e" is_pre_election: true
I20260924 07:10:09.561404 8062 raft_consensus.cc:2454] T 5d3a00f27fab4e0ba766328042632f12 P 83906483277942bab937e411fc4c666e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ba1e1457a494fc58f23c23c7e156a59 in term 0.
I20260924 07:10:09.561379 8202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3a00f27fab4e0ba766328042632f12" candidate_uuid: "3ba1e1457a494fc58f23c23c7e156a59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1865a617384ee3aa88bc78243d7949" is_pre_election: true
I20260924 07:10:09.561517 8202 raft_consensus.cc:2454] T 5d3a00f27fab4e0ba766328042632f12 P 5f1865a617384ee3aa88bc78243d7949 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ba1e1457a494fc58f23c23c7e156a59 in term 0.
I20260924 07:10:09.561573 8274 leader_election.cc:302] T 5d3a00f27fab4e0ba766328042632f12 P 3ba1e1457a494fc58f23c23c7e156a59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ba1e1457a494fc58f23c23c7e156a59, 83906483277942bab937e411fc4c666e; no voters:
I20260924 07:10:09.561693 8421 raft_consensus.cc:2781] T 5d3a00f27fab4e0ba766328042632f12 P 3ba1e1457a494fc58f23c23c7e156a59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:10:09.561746 8421 raft_consensus.cc:482] T 5d3a00f27fab4e0ba766328042632f12 P 3ba1e1457a494fc58f23c23c7e156a59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:10:09.561771 8421 raft_consensus.cc:3037] T 5d3a00f27fab4e0ba766328042632f12 P 3ba1e1457a494fc58f23c23c7e156a59 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:09.562436 8421 raft_consensus.cc:504] T 5d3a00f27fab4e0ba766328042632f12 P 3ba1e1457a494fc58f23c23c7e156a59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.562606 8421 leader_election.cc:288] T 5d3a00f27fab4e0ba766328042632f12 P 3ba1e1457a494fc58f23c23c7e156a59 [CANDIDATE]: Term 1 election: Requested vote from peers 83906483277942bab937e411fc4c666e (127.0.98.1:45771), 5f1865a617384ee3aa88bc78243d7949 (127.0.98.2:42777)
I20260924 07:10:09.562726 8062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3a00f27fab4e0ba766328042632f12" candidate_uuid: "3ba1e1457a494fc58f23c23c7e156a59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83906483277942bab937e411fc4c666e"
I20260924 07:10:09.562805 8062 raft_consensus.cc:3037] T 5d3a00f27fab4e0ba766328042632f12 P 83906483277942bab937e411fc4c666e [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:09.562747 8202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d3a00f27fab4e0ba766328042632f12" candidate_uuid: "3ba1e1457a494fc58f23c23c7e156a59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1865a617384ee3aa88bc78243d7949"
I20260924 07:10:09.562839 8202 raft_consensus.cc:3037] T 5d3a00f27fab4e0ba766328042632f12 P 5f1865a617384ee3aa88bc78243d7949 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:09.563411 8202 raft_consensus.cc:2454] T 5d3a00f27fab4e0ba766328042632f12 P 5f1865a617384ee3aa88bc78243d7949 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba1e1457a494fc58f23c23c7e156a59 in term 1.
I20260924 07:10:09.563423 8062 raft_consensus.cc:2454] T 5d3a00f27fab4e0ba766328042632f12 P 83906483277942bab937e411fc4c666e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba1e1457a494fc58f23c23c7e156a59 in term 1.
I20260924 07:10:09.563577 8274 leader_election.cc:302] T 5d3a00f27fab4e0ba766328042632f12 P 3ba1e1457a494fc58f23c23c7e156a59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ba1e1457a494fc58f23c23c7e156a59, 5f1865a617384ee3aa88bc78243d7949; no voters:
I20260924 07:10:09.563665 8421 raft_consensus.cc:2781] T 5d3a00f27fab4e0ba766328042632f12 P 3ba1e1457a494fc58f23c23c7e156a59 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:10:09.563726 8421 raft_consensus.cc:686] T 5d3a00f27fab4e0ba766328042632f12 P 3ba1e1457a494fc58f23c23c7e156a59 [term 1 LEADER]: Becoming Leader. State: Replica: 3ba1e1457a494fc58f23c23c7e156a59, State: Running, Role: LEADER
I20260924 07:10:09.563805 8421 consensus_queue.cc:237] T 5d3a00f27fab4e0ba766328042632f12 P 3ba1e1457a494fc58f23c23c7e156a59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.564083 8426 tablet_replica.cc:439] T 5d3a00f27fab4e0ba766328042632f12 P 3ba1e1457a494fc58f23c23c7e156a59: TxnStatusTablet state changed. Reason: New leader 3ba1e1457a494fc58f23c23c7e156a59. Latest consensus state: current_term: 1 leader_uuid: "3ba1e1457a494fc58f23c23c7e156a59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } } }
I20260924 07:10:09.564215 8426 tablet_replica.cc:442] T 5d3a00f27fab4e0ba766328042632f12 P 3ba1e1457a494fc58f23c23c7e156a59: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:10:09.564363 8460 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 07:10:09.564412 7905 catalog_manager.cc:5495] T 5d3a00f27fab4e0ba766328042632f12 P 3ba1e1457a494fc58f23c23c7e156a59 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ba1e1457a494fc58f23c23c7e156a59 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3ba1e1457a494fc58f23c23c7e156a59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } health_report { overall_health: HEALTHY } } }
I20260924 07:10:09.564504 8460 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 07:10:09.578073 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:09.575232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44545 (local address 127.0.0.1:39376)
0924 07:10:09.575311 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:09.575322 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:09.575376 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:09.575522 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:09.575522 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:09.575523 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:10:09.575563 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:09.575565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.576450 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.576451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:09.577126 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:09.577129 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.577643 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.577644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:09.577699 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:09.577712 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:10:09.577914 (+ 202us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:10:09.577924 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 07:10:09.577977 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 07:10:09.578017 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 07:10:09.578087 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:09.575257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39376 (local address 127.0.98.3:44545)
0924 07:10:09.575322 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:09.575325 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:09.575368 (+ 43us) server_negotiation.cc:407] Connection header received
0924 07:10:09.575439 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:09.575440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:09.575450 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:09.575518 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:10:09.575628 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.576363 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.577208 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.577506 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.577578 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:09.577800 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:10:09.577834 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:10:09.577889 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:09.577998 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 07:10:09.578019 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 07:10:09.579037 8062 raft_consensus.cc:1261] T 5d3a00f27fab4e0ba766328042632f12 P 83906483277942bab937e411fc4c666e [term 1 FOLLOWER]: Refusing update from remote peer 3ba1e1457a494fc58f23c23c7e156a59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:10:09.579034 8202 raft_consensus.cc:1261] T 5d3a00f27fab4e0ba766328042632f12 P 5f1865a617384ee3aa88bc78243d7949 [term 1 FOLLOWER]: Refusing update from remote peer 3ba1e1457a494fc58f23c23c7e156a59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:10:09.579250 8421 consensus_queue.cc:1035] T 5d3a00f27fab4e0ba766328042632f12 P 3ba1e1457a494fc58f23c23c7e156a59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:10:09.579330 8426 consensus_queue.cc:1035] T 5d3a00f27fab4e0ba766328042632f12 P 3ba1e1457a494fc58f23c23c7e156a59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:10:09.579902 8422 tablet_replica.cc:439] T 5d3a00f27fab4e0ba766328042632f12 P 83906483277942bab937e411fc4c666e: TxnStatusTablet state changed. Reason: New leader 3ba1e1457a494fc58f23c23c7e156a59. Latest consensus state: current_term: 1 leader_uuid: "3ba1e1457a494fc58f23c23c7e156a59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } } }
I20260924 07:10:09.579999 8422 tablet_replica.cc:442] T 5d3a00f27fab4e0ba766328042632f12 P 83906483277942bab937e411fc4c666e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:10:09.580132 8424 tablet_replica.cc:439] T 5d3a00f27fab4e0ba766328042632f12 P 5f1865a617384ee3aa88bc78243d7949: TxnStatusTablet state changed. Reason: New leader 3ba1e1457a494fc58f23c23c7e156a59. Latest consensus state: current_term: 1 leader_uuid: "3ba1e1457a494fc58f23c23c7e156a59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } } }
I20260924 07:10:09.580217 8424 tablet_replica.cc:442] T 5d3a00f27fab4e0ba766328042632f12 P 5f1865a617384ee3aa88bc78243d7949: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:10:09.580590 8426 tablet_replica.cc:439] T 5d3a00f27fab4e0ba766328042632f12 P 3ba1e1457a494fc58f23c23c7e156a59: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3ba1e1457a494fc58f23c23c7e156a59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } } }
I20260924 07:10:09.580673 8426 tablet_replica.cc:442] T 5d3a00f27fab4e0ba766328042632f12 P 3ba1e1457a494fc58f23c23c7e156a59: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:10:09.580969 8421 tablet_replica.cc:439] T 5d3a00f27fab4e0ba766328042632f12 P 3ba1e1457a494fc58f23c23c7e156a59: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3ba1e1457a494fc58f23c23c7e156a59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } } }
I20260924 07:10:09.581058 8421 tablet_replica.cc:442] T 5d3a00f27fab4e0ba766328042632f12 P 3ba1e1457a494fc58f23c23c7e156a59: This TxnStatusTablet replica's current role is: LEADER
I20260924 07:10:09.581167 8424 tablet_replica.cc:439] T 5d3a00f27fab4e0ba766328042632f12 P 5f1865a617384ee3aa88bc78243d7949: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3ba1e1457a494fc58f23c23c7e156a59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } } }
I20260924 07:10:09.581238 8424 tablet_replica.cc:442] T 5d3a00f27fab4e0ba766328042632f12 P 5f1865a617384ee3aa88bc78243d7949: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:10:09.581543 8422 tablet_replica.cc:439] T 5d3a00f27fab4e0ba766328042632f12 P 83906483277942bab937e411fc4c666e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3ba1e1457a494fc58f23c23c7e156a59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } } }
I20260924 07:10:09.581627 8422 tablet_replica.cc:442] T 5d3a00f27fab4e0ba766328042632f12 P 83906483277942bab937e411fc4c666e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 07:10:09.585376 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:09.582250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43644 (local address 127.0.98.1:45771)
0924 07:10:09.582288 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:09.582289 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:09.582516 (+ 227us) server_negotiation.cc:407] Connection header received
0924 07:10:09.582554 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:09.582555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:09.582574 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:09.582644 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:10:09.582730 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.583472 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.584298 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.584581 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.584654 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:09.584972 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:10:09.585113 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:10:09.585193 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:09.585265 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 07:10:09.585284 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 07:10:09.590711 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:09.587798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44545 (local address 127.0.98.1:38981)
0924 07:10:09.587964 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:09.587975 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:09.588038 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:09.588169 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:09.588170 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:09.588171 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:10:09.588237 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:09.588238 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.588937 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.588938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:09.589915 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:09.589918 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.590278 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.590279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:09.590347 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:09.590361 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:10:09.590524 (+ 163us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:10:09.590526 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 07:10:09.590580 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 07:10:09.590624 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":88,"threads_started":1}
I20260924 07:10:09.590701 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:09.587889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38981 (local address 127.0.98.3:44545)
0924 07:10:09.587930 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:09.587932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:09.588023 (+ 91us) server_negotiation.cc:407] Connection header received
0924 07:10:09.588095 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:09.588095 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:09.588103 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:09.588152 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:10:09.588323 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.588862 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.590005 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.590194 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.590260 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:09.590441 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:10:09.590465 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:10:09.590505 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:09.590588 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 07:10:09.590606 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 07:10:09.592911 8202 raft_consensus.cc:1261] T f388e29619ba4477b5c48013269e059e P 5f1865a617384ee3aa88bc78243d7949 [term 1 FOLLOWER]: Refusing update from remote peer 83906483277942bab937e411fc4c666e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:10:09.593019 8337 raft_consensus.cc:1261] T f388e29619ba4477b5c48013269e059e P 3ba1e1457a494fc58f23c23c7e156a59 [term 1 FOLLOWER]: Refusing update from remote peer 83906483277942bab937e411fc4c666e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:10:09.593132 8422 consensus_queue.cc:1035] T f388e29619ba4477b5c48013269e059e P 83906483277942bab937e411fc4c666e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:10:09.593240 8448 consensus_queue.cc:1035] T f388e29619ba4477b5c48013269e059e P 83906483277942bab937e411fc4c666e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:10:09.594487 8430 mvcc.cc:205] Tried to move back new op lower bound from 7332797684090736640 to 7332797683545395200. Current Snapshot: MvccSnapshot[applied={T|T < 7332797684090736640}]
I20260924 07:10:09.601747 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:09.598608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36269 (local address 127.0.98.1:45771)
0924 07:10:09.598647 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:09.598649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:09.598798 (+ 149us) server_negotiation.cc:407] Connection header received
0924 07:10:09.598838 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:09.598838 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:09.598848 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:09.598904 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:10:09.598981 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.599700 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.600702 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.601043 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.601109 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:09.601305 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:10:09.601339 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:10:09.601365 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:09.601650 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 07:10:09.601674 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 07:10:09.602511 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:09.598554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45771 (local address 127.0.98.3:36269)
0924 07:10:09.598718 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:09.598729 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:09.598789 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:09.598909 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:09.598909 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:09.598910 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 07:10:09.598944 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:09.598945 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.599812 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.599813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:09.600624 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:09.600626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.601142 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.601143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:09.601201 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:09.601215 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 07:10:09.601404 (+ 189us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 07:10:09.601406 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 07:10:09.602355 (+ 949us) client_negotiation.cc:240] Negotiation successful
0924 07:10:09.602422 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
I20260924 07:10:09.610369 7904 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:10:09.610476 7904 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:10:09.611119 7904 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32fa0114c29249baacf9a200be709192: Sending DeleteTablet for 3 replicas of tablet f388e29619ba4477b5c48013269e059e
I20260924 07:10:09.611405 8322 tablet_service.cc:1514] Processing DeleteTablet for tablet f388e29619ba4477b5c48013269e059e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:10:09 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39354
I20260924 07:10:09.611502 8438 tablet_replica.cc:330] T f388e29619ba4477b5c48013269e059e P 3ba1e1457a494fc58f23c23c7e156a59: stopping tablet replica
I20260924 07:10:09.611509 8182 tablet_service.cc:1514] Processing DeleteTablet for tablet f388e29619ba4477b5c48013269e059e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:10:09 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47584
I20260924 07:10:09.611544 8438 raft_consensus.cc:2229] T f388e29619ba4477b5c48013269e059e P 3ba1e1457a494fc58f23c23c7e156a59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:10:09.611579 8438 raft_consensus.cc:2258] T f388e29619ba4477b5c48013269e059e P 3ba1e1457a494fc58f23c23c7e156a59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:10:09.611631 8439 tablet_replica.cc:330] T f388e29619ba4477b5c48013269e059e P 5f1865a617384ee3aa88bc78243d7949: stopping tablet replica
I20260924 07:10:09.611651 8042 tablet_service.cc:1514] Processing DeleteTablet for tablet f388e29619ba4477b5c48013269e059e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:10:09 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43628
I20260924 07:10:09.611688 8439 raft_consensus.cc:2229] T f388e29619ba4477b5c48013269e059e P 5f1865a617384ee3aa88bc78243d7949 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:10:09.611727 8439 raft_consensus.cc:2258] T f388e29619ba4477b5c48013269e059e P 5f1865a617384ee3aa88bc78243d7949 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:10:09.611815 8440 tablet_replica.cc:330] T f388e29619ba4477b5c48013269e059e P 83906483277942bab937e411fc4c666e: stopping tablet replica
I20260924 07:10:09.611866 8440 raft_consensus.cc:2229] T f388e29619ba4477b5c48013269e059e P 83906483277942bab937e411fc4c666e [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:10:09.611948 8440 raft_consensus.cc:2258] T f388e29619ba4477b5c48013269e059e P 83906483277942bab937e411fc4c666e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:10:09.612308 8440 ts_tablet_manager.cc:1905] T f388e29619ba4477b5c48013269e059e P 83906483277942bab937e411fc4c666e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:10:09.612390 8439 ts_tablet_manager.cc:1905] T f388e29619ba4477b5c48013269e059e P 5f1865a617384ee3aa88bc78243d7949: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:10:09.614096 8438 ts_tablet_manager.cc:1905] T f388e29619ba4477b5c48013269e059e P 3ba1e1457a494fc58f23c23c7e156a59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:10:09.614516 8439 ts_tablet_manager.cc:1918] T f388e29619ba4477b5c48013269e059e P 5f1865a617384ee3aa88bc78243d7949: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:10:09.614562 8439 log.cc:1196] T f388e29619ba4477b5c48013269e059e P 5f1865a617384ee3aa88bc78243d7949: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-1/wal/wals/f388e29619ba4477b5c48013269e059e
I20260924 07:10:09.614784 8439 ts_tablet_manager.cc:1939] T f388e29619ba4477b5c48013269e059e P 5f1865a617384ee3aa88bc78243d7949: Deleting consensus metadata
I20260924 07:10:09.615078 7890 catalog_manager.cc:4841] TS 5f1865a617384ee3aa88bc78243d7949 (127.0.98.2:42777): tablet f388e29619ba4477b5c48013269e059e (table test-table [id=9c27dfa2be84473e807bc42a3db794e1]) successfully deleted
I20260924 07:10:09.615859 8440 ts_tablet_manager.cc:1918] T f388e29619ba4477b5c48013269e059e P 83906483277942bab937e411fc4c666e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:10:09.615914 8440 log.cc:1196] T f388e29619ba4477b5c48013269e059e P 83906483277942bab937e411fc4c666e: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-0/wal/wals/f388e29619ba4477b5c48013269e059e
I20260924 07:10:09.616119 8440 ts_tablet_manager.cc:1939] T f388e29619ba4477b5c48013269e059e P 83906483277942bab937e411fc4c666e: Deleting consensus metadata
I20260924 07:10:09.616159 8438 ts_tablet_manager.cc:1918] T f388e29619ba4477b5c48013269e059e P 3ba1e1457a494fc58f23c23c7e156a59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:10:09.616210 8438 log.cc:1196] T f388e29619ba4477b5c48013269e059e P 3ba1e1457a494fc58f23c23c7e156a59: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-2/wal/wals/f388e29619ba4477b5c48013269e059e
I20260924 07:10:09.616420 8438 ts_tablet_manager.cc:1939] T f388e29619ba4477b5c48013269e059e P 3ba1e1457a494fc58f23c23c7e156a59: Deleting consensus metadata
I20260924 07:10:09.616860 7890 catalog_manager.cc:4841] TS 83906483277942bab937e411fc4c666e (127.0.98.1:45771): tablet f388e29619ba4477b5c48013269e059e (table test-table [id=9c27dfa2be84473e807bc42a3db794e1]) successfully deleted
I20260924 07:10:09.616941 7889 catalog_manager.cc:4841] TS 3ba1e1457a494fc58f23c23c7e156a59 (127.0.98.3:44545): tablet f388e29619ba4477b5c48013269e059e (table test-table [id=9c27dfa2be84473e807bc42a3db794e1]) successfully deleted
Sep 24 07:10:09 dist-test-slave-14ww krb5kdc[7860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 07:10:09 dist-test-slave-14ww krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:09 dist-test-slave-14ww krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 07:10:09.641430 7954 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:09.634781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33486 (local address 127.0.98.62:34025)
0924 07:10:09.634824 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:09.634827 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:09.634937 (+ 110us) server_negotiation.cc:407] Connection header received
0924 07:10:09.634995 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:09.634996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:09.635007 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:09.635077 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:09.635789 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.636576 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.637593 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.637906 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.637981 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:09.641356 (+ 3375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33486: BlockingRecv error: recv got EOF from 127.0.0.1:33486 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 24 07:10:09 dist-test-slave-14ww krb5kdc[7860](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 07:10:09.651043 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:09.642398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33488 (local address 127.0.98.62:34025)
0924 07:10:09.642434 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:09.642436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:09.642690 (+ 254us) server_negotiation.cc:407] Connection header received
0924 07:10:09.642741 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:09.642742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:09.642752 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:09.642814 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:09.643403 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.644167 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.645170 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.645464 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.645537 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:09.648155 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:09.648156 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:09.648156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:09.648164 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:09.649851 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:09.650240 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:09.650241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:09.650242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:09.650302 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:09.650519 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:09.650520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:09.650521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:09.650646 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:09.650751 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:09.650874 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 07:10:09.650948 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 07:10:09.652665 7904 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:10:09.652884 7904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:10:09.654435 8042 tablet_service.cc:1467] Processing CreateTablet for tablet 51a903b7a98840dc8a01049591d1478f (DEFAULT_TABLE table=test-table [id=92ab8b65846e4d4ca16d797388fcc538]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:10:09.654572 8042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51a903b7a98840dc8a01049591d1478f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:09.656062 8182 tablet_service.cc:1467] Processing CreateTablet for tablet 51a903b7a98840dc8a01049591d1478f (DEFAULT_TABLE table=test-table [id=92ab8b65846e4d4ca16d797388fcc538]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:10:09.656190 8182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51a903b7a98840dc8a01049591d1478f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:09.656806 8417 tablet_bootstrap.cc:492] T 51a903b7a98840dc8a01049591d1478f P 83906483277942bab937e411fc4c666e: Bootstrap starting.
I20260924 07:10:09.656898 8322 tablet_service.cc:1467] Processing CreateTablet for tablet 51a903b7a98840dc8a01049591d1478f (DEFAULT_TABLE table=test-table [id=92ab8b65846e4d4ca16d797388fcc538]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:10:09.657018 8322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51a903b7a98840dc8a01049591d1478f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:09.657284 8417 tablet_bootstrap.cc:654] T 51a903b7a98840dc8a01049591d1478f P 83906483277942bab937e411fc4c666e: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:09.658267 8416 tablet_bootstrap.cc:492] T 51a903b7a98840dc8a01049591d1478f P 3ba1e1457a494fc58f23c23c7e156a59: Bootstrap starting.
I20260924 07:10:09.658416 8420 tablet_bootstrap.cc:492] T 51a903b7a98840dc8a01049591d1478f P 5f1865a617384ee3aa88bc78243d7949: Bootstrap starting.
I20260924 07:10:09.658538 8416 tablet_bootstrap.cc:654] T 51a903b7a98840dc8a01049591d1478f P 3ba1e1457a494fc58f23c23c7e156a59: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:09.659140 8420 tablet_bootstrap.cc:654] T 51a903b7a98840dc8a01049591d1478f P 5f1865a617384ee3aa88bc78243d7949: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:09.659798 8416 tablet_bootstrap.cc:492] T 51a903b7a98840dc8a01049591d1478f P 3ba1e1457a494fc58f23c23c7e156a59: No bootstrap required, opened a new log
I20260924 07:10:09.659853 8416 ts_tablet_manager.cc:1397] T 51a903b7a98840dc8a01049591d1478f P 3ba1e1457a494fc58f23c23c7e156a59: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 07:10:09.660010 8416 raft_consensus.cc:348] T 51a903b7a98840dc8a01049591d1478f P 3ba1e1457a494fc58f23c23c7e156a59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.660091 8416 raft_consensus.cc:374] T 51a903b7a98840dc8a01049591d1478f P 3ba1e1457a494fc58f23c23c7e156a59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:09.660115 8416 raft_consensus.cc:729] T 51a903b7a98840dc8a01049591d1478f P 3ba1e1457a494fc58f23c23c7e156a59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ba1e1457a494fc58f23c23c7e156a59, State: Initialized, Role: FOLLOWER
I20260924 07:10:09.660173 8416 consensus_queue.cc:260] T 51a903b7a98840dc8a01049591d1478f P 3ba1e1457a494fc58f23c23c7e156a59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.660269 8416 ts_tablet_manager.cc:1428] T 51a903b7a98840dc8a01049591d1478f P 3ba1e1457a494fc58f23c23c7e156a59: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:09.660596 8420 tablet_bootstrap.cc:492] T 51a903b7a98840dc8a01049591d1478f P 5f1865a617384ee3aa88bc78243d7949: No bootstrap required, opened a new log
I20260924 07:10:09.660642 8420 ts_tablet_manager.cc:1397] T 51a903b7a98840dc8a01049591d1478f P 5f1865a617384ee3aa88bc78243d7949: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:10:09.660820 8420 raft_consensus.cc:348] T 51a903b7a98840dc8a01049591d1478f P 5f1865a617384ee3aa88bc78243d7949 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.660895 8420 raft_consensus.cc:374] T 51a903b7a98840dc8a01049591d1478f P 5f1865a617384ee3aa88bc78243d7949 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:09.660923 8420 raft_consensus.cc:729] T 51a903b7a98840dc8a01049591d1478f P 5f1865a617384ee3aa88bc78243d7949 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f1865a617384ee3aa88bc78243d7949, State: Initialized, Role: FOLLOWER
I20260924 07:10:09.660962 8420 consensus_queue.cc:260] T 51a903b7a98840dc8a01049591d1478f P 5f1865a617384ee3aa88bc78243d7949 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.661051 8420 ts_tablet_manager.cc:1428] T 51a903b7a98840dc8a01049591d1478f P 5f1865a617384ee3aa88bc78243d7949: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 07:10:09.665169 8417 tablet_bootstrap.cc:492] T 51a903b7a98840dc8a01049591d1478f P 83906483277942bab937e411fc4c666e: No bootstrap required, opened a new log
I20260924 07:10:09.665228 8417 ts_tablet_manager.cc:1397] T 51a903b7a98840dc8a01049591d1478f P 83906483277942bab937e411fc4c666e: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260924 07:10:09.665354 8417 raft_consensus.cc:348] T 51a903b7a98840dc8a01049591d1478f P 83906483277942bab937e411fc4c666e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.665423 8417 raft_consensus.cc:374] T 51a903b7a98840dc8a01049591d1478f P 83906483277942bab937e411fc4c666e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:09.665444 8417 raft_consensus.cc:729] T 51a903b7a98840dc8a01049591d1478f P 83906483277942bab937e411fc4c666e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83906483277942bab937e411fc4c666e, State: Initialized, Role: FOLLOWER
I20260924 07:10:09.665484 8417 consensus_queue.cc:260] T 51a903b7a98840dc8a01049591d1478f P 83906483277942bab937e411fc4c666e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.665568 8417 ts_tablet_manager.cc:1428] T 51a903b7a98840dc8a01049591d1478f P 83906483277942bab937e411fc4c666e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 07:10:09.733626 8421 raft_consensus.cc:482] T 51a903b7a98840dc8a01049591d1478f P 3ba1e1457a494fc58f23c23c7e156a59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:10:09.733738 8421 raft_consensus.cc:504] T 51a903b7a98840dc8a01049591d1478f P 3ba1e1457a494fc58f23c23c7e156a59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.733909 8421 leader_election.cc:288] T 51a903b7a98840dc8a01049591d1478f P 3ba1e1457a494fc58f23c23c7e156a59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83906483277942bab937e411fc4c666e (127.0.98.1:45771), 5f1865a617384ee3aa88bc78243d7949 (127.0.98.2:42777)
I20260924 07:10:09.734134 8202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51a903b7a98840dc8a01049591d1478f" candidate_uuid: "3ba1e1457a494fc58f23c23c7e156a59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1865a617384ee3aa88bc78243d7949" is_pre_election: true
I20260924 07:10:09.734159 8062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51a903b7a98840dc8a01049591d1478f" candidate_uuid: "3ba1e1457a494fc58f23c23c7e156a59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83906483277942bab937e411fc4c666e" is_pre_election: true
I20260924 07:10:09.734234 8062 raft_consensus.cc:2454] T 51a903b7a98840dc8a01049591d1478f P 83906483277942bab937e411fc4c666e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ba1e1457a494fc58f23c23c7e156a59 in term 0.
I20260924 07:10:09.734234 8202 raft_consensus.cc:2454] T 51a903b7a98840dc8a01049591d1478f P 5f1865a617384ee3aa88bc78243d7949 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ba1e1457a494fc58f23c23c7e156a59 in term 0.
I20260924 07:10:09.734402 8274 leader_election.cc:302] T 51a903b7a98840dc8a01049591d1478f P 3ba1e1457a494fc58f23c23c7e156a59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ba1e1457a494fc58f23c23c7e156a59, 83906483277942bab937e411fc4c666e; no voters:
I20260924 07:10:09.734500 8421 raft_consensus.cc:2781] T 51a903b7a98840dc8a01049591d1478f P 3ba1e1457a494fc58f23c23c7e156a59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:10:09.734539 8421 raft_consensus.cc:482] T 51a903b7a98840dc8a01049591d1478f P 3ba1e1457a494fc58f23c23c7e156a59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:10:09.734566 8421 raft_consensus.cc:3037] T 51a903b7a98840dc8a01049591d1478f P 3ba1e1457a494fc58f23c23c7e156a59 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:09.735242 8421 raft_consensus.cc:504] T 51a903b7a98840dc8a01049591d1478f P 3ba1e1457a494fc58f23c23c7e156a59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.735373 8421 leader_election.cc:288] T 51a903b7a98840dc8a01049591d1478f P 3ba1e1457a494fc58f23c23c7e156a59 [CANDIDATE]: Term 1 election: Requested vote from peers 83906483277942bab937e411fc4c666e (127.0.98.1:45771), 5f1865a617384ee3aa88bc78243d7949 (127.0.98.2:42777)
I20260924 07:10:09.735513 8062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51a903b7a98840dc8a01049591d1478f" candidate_uuid: "3ba1e1457a494fc58f23c23c7e156a59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83906483277942bab937e411fc4c666e"
I20260924 07:10:09.735546 8202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51a903b7a98840dc8a01049591d1478f" candidate_uuid: "3ba1e1457a494fc58f23c23c7e156a59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1865a617384ee3aa88bc78243d7949"
I20260924 07:10:09.735594 8062 raft_consensus.cc:3037] T 51a903b7a98840dc8a01049591d1478f P 83906483277942bab937e411fc4c666e [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:09.735605 8202 raft_consensus.cc:3037] T 51a903b7a98840dc8a01049591d1478f P 5f1865a617384ee3aa88bc78243d7949 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:09.736316 8202 raft_consensus.cc:2454] T 51a903b7a98840dc8a01049591d1478f P 5f1865a617384ee3aa88bc78243d7949 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba1e1457a494fc58f23c23c7e156a59 in term 1.
I20260924 07:10:09.736328 8062 raft_consensus.cc:2454] T 51a903b7a98840dc8a01049591d1478f P 83906483277942bab937e411fc4c666e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ba1e1457a494fc58f23c23c7e156a59 in term 1.
I20260924 07:10:09.736495 8274 leader_election.cc:302] T 51a903b7a98840dc8a01049591d1478f P 3ba1e1457a494fc58f23c23c7e156a59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ba1e1457a494fc58f23c23c7e156a59, 5f1865a617384ee3aa88bc78243d7949; no voters:
I20260924 07:10:09.736649 8421 raft_consensus.cc:2781] T 51a903b7a98840dc8a01049591d1478f P 3ba1e1457a494fc58f23c23c7e156a59 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:10:09.736716 8421 raft_consensus.cc:686] T 51a903b7a98840dc8a01049591d1478f P 3ba1e1457a494fc58f23c23c7e156a59 [term 1 LEADER]: Becoming Leader. State: Replica: 3ba1e1457a494fc58f23c23c7e156a59, State: Running, Role: LEADER
I20260924 07:10:09.736797 8421 consensus_queue.cc:237] T 51a903b7a98840dc8a01049591d1478f P 3ba1e1457a494fc58f23c23c7e156a59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.737385 7904 catalog_manager.cc:5495] T 51a903b7a98840dc8a01049591d1478f P 3ba1e1457a494fc58f23c23c7e156a59 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ba1e1457a494fc58f23c23c7e156a59 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3ba1e1457a494fc58f23c23c7e156a59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } health_report { overall_health: HEALTHY } } }
I20260924 07:10:09.751524 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:09.749301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39386 (local address 127.0.98.3:44545)
0924 07:10:09.749359 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:09.749361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:09.749370 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:10:09.749393 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:09.749394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:09.749410 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:09.749461 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:10:09.749635 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.750089 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.750889 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.751067 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.751122 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:09.751297 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:10:09.751329 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:10:09.751379 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:09.751442 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 07:10:09.751459 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 07:10:09.752303 8062 raft_consensus.cc:1261] T 51a903b7a98840dc8a01049591d1478f P 83906483277942bab937e411fc4c666e [term 1 FOLLOWER]: Refusing update from remote peer 3ba1e1457a494fc58f23c23c7e156a59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:10:09.752326 8202 raft_consensus.cc:1261] T 51a903b7a98840dc8a01049591d1478f P 5f1865a617384ee3aa88bc78243d7949 [term 1 FOLLOWER]: Refusing update from remote peer 3ba1e1457a494fc58f23c23c7e156a59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:10:09.752528 8426 consensus_queue.cc:1035] T 51a903b7a98840dc8a01049591d1478f P 3ba1e1457a494fc58f23c23c7e156a59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:10:09.752676 8421 consensus_queue.cc:1035] T 51a903b7a98840dc8a01049591d1478f P 3ba1e1457a494fc58f23c23c7e156a59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:10:09.753957 8429 mvcc.cc:205] Tried to move back new op lower bound from 7332797684743872512 to 7332797684682194944. Current Snapshot: MvccSnapshot[applied={T|T < 7332797684743872512 or (T in {7332797684743872512})}]
I20260924 07:10:09.755530 7905 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:10:09.755623 7905 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:10:09.756201 7905 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32fa0114c29249baacf9a200be709192: Sending DeleteTablet for 3 replicas of tablet 51a903b7a98840dc8a01049591d1478f
I20260924 07:10:09.756455 8322 tablet_service.cc:1514] Processing DeleteTablet for tablet 51a903b7a98840dc8a01049591d1478f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:10:09 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39354
I20260924 07:10:09.756456 8042 tablet_service.cc:1514] Processing DeleteTablet for tablet 51a903b7a98840dc8a01049591d1478f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:10:09 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43628
I20260924 07:10:09.756469 8182 tablet_service.cc:1514] Processing DeleteTablet for tablet 51a903b7a98840dc8a01049591d1478f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:10:09 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47584
I20260924 07:10:09.756538 8439 tablet_replica.cc:330] T 51a903b7a98840dc8a01049591d1478f P 5f1865a617384ee3aa88bc78243d7949: stopping tablet replica
I20260924 07:10:09.756538 8440 tablet_replica.cc:330] T 51a903b7a98840dc8a01049591d1478f P 83906483277942bab937e411fc4c666e: stopping tablet replica
I20260924 07:10:09.756582 8439 raft_consensus.cc:2229] T 51a903b7a98840dc8a01049591d1478f P 5f1865a617384ee3aa88bc78243d7949 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:10:09.756587 8440 raft_consensus.cc:2229] T 51a903b7a98840dc8a01049591d1478f P 83906483277942bab937e411fc4c666e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:10:09.756623 8440 raft_consensus.cc:2258] T 51a903b7a98840dc8a01049591d1478f P 83906483277942bab937e411fc4c666e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:10:09.756639 8439 pending_rounds.cc:62] T 51a903b7a98840dc8a01049591d1478f P 5f1865a617384ee3aa88bc78243d7949: Trying to abort 2 pending ops.
I20260924 07:10:09.756664 8439 pending_rounds.cc:68] T 51a903b7a98840dc8a01049591d1478f P 5f1865a617384ee3aa88bc78243d7949: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7332797684743872512 op_type: WRITE_OP write_request { tablet_id: "51a903b7a98840dc8a01049591d1478f" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\375\233\323\325\006\"8\n\ntest-admin\022*\n 92ab8b65846e4d4ca16d797388fcc538\020\001\030\001 \001(\001" signature: "<redacted>""$\267\251#\013j4\022aP\362\275+\364\030\003\365\003$3q\004\231sk\215\353\225\320\321\254\352\022-,\236\256\227%\237\342E<J\355R\215\317\353\251(G\232i\004\004\315|\312\274\221\227\205N" signing_key_seq_num: 0 } } request_id { client_id: "6134777cc2ec402fbb6c82d449eb8bd4" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260924 07:10:09.756867 8439 pending_rounds.cc:68] T 51a903b7a98840dc8a01049591d1478f P 5f1865a617384ee3aa88bc78243d7949: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332797684682194944 op_type: NO_OP noop_request { }
I20260924 07:10:09.756907 8439 raft_consensus.cc:2866] T 51a903b7a98840dc8a01049591d1478f P 5f1865a617384ee3aa88bc78243d7949 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260924 07:10:09.756917 8440 ts_tablet_manager.cc:1905] T 51a903b7a98840dc8a01049591d1478f P 83906483277942bab937e411fc4c666e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:10:09.756933 8439 raft_consensus.cc:2258] T 51a903b7a98840dc8a01049591d1478f P 5f1865a617384ee3aa88bc78243d7949 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:10:09.757011 8438 tablet_replica.cc:330] T 51a903b7a98840dc8a01049591d1478f P 3ba1e1457a494fc58f23c23c7e156a59: stopping tablet replica
I20260924 07:10:09.757071 8438 raft_consensus.cc:2229] T 51a903b7a98840dc8a01049591d1478f P 3ba1e1457a494fc58f23c23c7e156a59 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:10:09.757117 8438 raft_consensus.cc:2258] T 51a903b7a98840dc8a01049591d1478f P 3ba1e1457a494fc58f23c23c7e156a59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:10:09.757493 8438 ts_tablet_manager.cc:1905] T 51a903b7a98840dc8a01049591d1478f P 3ba1e1457a494fc58f23c23c7e156a59: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 07:10:09.757838 8494 consensus_queue.cc:1159] T 51a903b7a98840dc8a01049591d1478f P 5f1865a617384ee3aa88bc78243d7949 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "5f1865a617384ee3aa88bc78243d7949" status { last_received { term: 1 index: 2 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 2 } }
I20260924 07:10:09.758564 8440 ts_tablet_manager.cc:1918] T 51a903b7a98840dc8a01049591d1478f P 83906483277942bab937e411fc4c666e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:10:09.758635 8440 log.cc:1196] T 51a903b7a98840dc8a01049591d1478f P 83906483277942bab937e411fc4c666e: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-0/wal/wals/51a903b7a98840dc8a01049591d1478f
I20260924 07:10:09.758853 8440 ts_tablet_manager.cc:1939] T 51a903b7a98840dc8a01049591d1478f P 83906483277942bab937e411fc4c666e: Deleting consensus metadata
I20260924 07:10:09.759008 8438 ts_tablet_manager.cc:1918] T 51a903b7a98840dc8a01049591d1478f P 3ba1e1457a494fc58f23c23c7e156a59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:10:09.759068 8438 log.cc:1196] T 51a903b7a98840dc8a01049591d1478f P 3ba1e1457a494fc58f23c23c7e156a59: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-2/wal/wals/51a903b7a98840dc8a01049591d1478f
I20260924 07:10:09.759212 7890 catalog_manager.cc:4841] TS 83906483277942bab937e411fc4c666e (127.0.98.1:45771): tablet 51a903b7a98840dc8a01049591d1478f (table test-table [id=92ab8b65846e4d4ca16d797388fcc538]) successfully deleted
I20260924 07:10:09.759281 8438 ts_tablet_manager.cc:1939] T 51a903b7a98840dc8a01049591d1478f P 3ba1e1457a494fc58f23c23c7e156a59: Deleting consensus metadata
I20260924 07:10:09.759567 7889 catalog_manager.cc:4841] TS 3ba1e1457a494fc58f23c23c7e156a59 (127.0.98.3:44545): tablet 51a903b7a98840dc8a01049591d1478f (table test-table [id=92ab8b65846e4d4ca16d797388fcc538]) successfully deleted
I20260924 07:10:09.759603 7905 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:10:09.759773 7905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:10:09.759799 8439 ts_tablet_manager.cc:1905] T 51a903b7a98840dc8a01049591d1478f P 5f1865a617384ee3aa88bc78243d7949: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:10:09.761102 8322 tablet_service.cc:1467] Processing CreateTablet for tablet e25ba7ddc62843ee9409a52a1ac4e23e (DEFAULT_TABLE table=test-table [id=deea0a214942429bbb09c78a0d52a3dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:10:09.761111 8042 tablet_service.cc:1467] Processing CreateTablet for tablet e25ba7ddc62843ee9409a52a1ac4e23e (DEFAULT_TABLE table=test-table [id=deea0a214942429bbb09c78a0d52a3dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:10:09.761224 8042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e25ba7ddc62843ee9409a52a1ac4e23e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:09.761231 8322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e25ba7ddc62843ee9409a52a1ac4e23e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:09.761365 8439 ts_tablet_manager.cc:1918] T 51a903b7a98840dc8a01049591d1478f P 5f1865a617384ee3aa88bc78243d7949: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:10:09.761420 8439 log.cc:1196] T 51a903b7a98840dc8a01049591d1478f P 5f1865a617384ee3aa88bc78243d7949: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-1/wal/wals/51a903b7a98840dc8a01049591d1478f
I20260924 07:10:09.761623 8439 ts_tablet_manager.cc:1939] T 51a903b7a98840dc8a01049591d1478f P 5f1865a617384ee3aa88bc78243d7949: Deleting consensus metadata
I20260924 07:10:09.761766 8182 tablet_service.cc:1467] Processing CreateTablet for tablet e25ba7ddc62843ee9409a52a1ac4e23e (DEFAULT_TABLE table=test-table [id=deea0a214942429bbb09c78a0d52a3dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:10:09.761852 8182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e25ba7ddc62843ee9409a52a1ac4e23e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:09.762425 8417 tablet_bootstrap.cc:492] T e25ba7ddc62843ee9409a52a1ac4e23e P 83906483277942bab937e411fc4c666e: Bootstrap starting.
I20260924 07:10:09.762739 8417 tablet_bootstrap.cc:654] T e25ba7ddc62843ee9409a52a1ac4e23e P 83906483277942bab937e411fc4c666e: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:09.763054 8420 tablet_bootstrap.cc:492] T e25ba7ddc62843ee9409a52a1ac4e23e P 5f1865a617384ee3aa88bc78243d7949: Bootstrap starting.
I20260924 07:10:09.763068 8416 tablet_bootstrap.cc:492] T e25ba7ddc62843ee9409a52a1ac4e23e P 3ba1e1457a494fc58f23c23c7e156a59: Bootstrap starting.
I20260924 07:10:09.763314 8417 tablet_bootstrap.cc:492] T e25ba7ddc62843ee9409a52a1ac4e23e P 83906483277942bab937e411fc4c666e: No bootstrap required, opened a new log
I20260924 07:10:09.763341 8416 tablet_bootstrap.cc:654] T e25ba7ddc62843ee9409a52a1ac4e23e P 3ba1e1457a494fc58f23c23c7e156a59: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:09.763360 8417 ts_tablet_manager.cc:1397] T e25ba7ddc62843ee9409a52a1ac4e23e P 83906483277942bab937e411fc4c666e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 07:10:09.763482 8420 tablet_bootstrap.cc:654] T e25ba7ddc62843ee9409a52a1ac4e23e P 5f1865a617384ee3aa88bc78243d7949: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:09.763469 8417 raft_consensus.cc:348] T e25ba7ddc62843ee9409a52a1ac4e23e P 83906483277942bab937e411fc4c666e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.763532 8417 raft_consensus.cc:374] T e25ba7ddc62843ee9409a52a1ac4e23e P 83906483277942bab937e411fc4c666e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:09.763561 8417 raft_consensus.cc:729] T e25ba7ddc62843ee9409a52a1ac4e23e P 83906483277942bab937e411fc4c666e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83906483277942bab937e411fc4c666e, State: Initialized, Role: FOLLOWER
I20260924 07:10:09.763610 8417 consensus_queue.cc:260] T e25ba7ddc62843ee9409a52a1ac4e23e P 83906483277942bab937e411fc4c666e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.763717 8417 ts_tablet_manager.cc:1428] T e25ba7ddc62843ee9409a52a1ac4e23e P 83906483277942bab937e411fc4c666e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:09.763789 8416 tablet_bootstrap.cc:492] T e25ba7ddc62843ee9409a52a1ac4e23e P 3ba1e1457a494fc58f23c23c7e156a59: No bootstrap required, opened a new log
I20260924 07:10:09.763849 8416 ts_tablet_manager.cc:1397] T e25ba7ddc62843ee9409a52a1ac4e23e P 3ba1e1457a494fc58f23c23c7e156a59: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 07:10:09.763957 8416 raft_consensus.cc:348] T e25ba7ddc62843ee9409a52a1ac4e23e P 3ba1e1457a494fc58f23c23c7e156a59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.764032 8416 raft_consensus.cc:374] T e25ba7ddc62843ee9409a52a1ac4e23e P 3ba1e1457a494fc58f23c23c7e156a59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:09.764055 8416 raft_consensus.cc:729] T e25ba7ddc62843ee9409a52a1ac4e23e P 3ba1e1457a494fc58f23c23c7e156a59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ba1e1457a494fc58f23c23c7e156a59, State: Initialized, Role: FOLLOWER
I20260924 07:10:09.764093 8416 consensus_queue.cc:260] T e25ba7ddc62843ee9409a52a1ac4e23e P 3ba1e1457a494fc58f23c23c7e156a59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.764209 8416 ts_tablet_manager.cc:1428] T e25ba7ddc62843ee9409a52a1ac4e23e P 3ba1e1457a494fc58f23c23c7e156a59: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:09.764472 7890 catalog_manager.cc:4841] TS 5f1865a617384ee3aa88bc78243d7949 (127.0.98.2:42777): tablet 51a903b7a98840dc8a01049591d1478f (table test-table [id=92ab8b65846e4d4ca16d797388fcc538]) successfully deleted
I20260924 07:10:09.764688 8420 tablet_bootstrap.cc:492] T e25ba7ddc62843ee9409a52a1ac4e23e P 5f1865a617384ee3aa88bc78243d7949: No bootstrap required, opened a new log
I20260924 07:10:09.764765 8420 ts_tablet_manager.cc:1397] T e25ba7ddc62843ee9409a52a1ac4e23e P 5f1865a617384ee3aa88bc78243d7949: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:10:09.764922 8420 raft_consensus.cc:348] T e25ba7ddc62843ee9409a52a1ac4e23e P 5f1865a617384ee3aa88bc78243d7949 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.764997 8420 raft_consensus.cc:374] T e25ba7ddc62843ee9409a52a1ac4e23e P 5f1865a617384ee3aa88bc78243d7949 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:09.765027 8420 raft_consensus.cc:729] T e25ba7ddc62843ee9409a52a1ac4e23e P 5f1865a617384ee3aa88bc78243d7949 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f1865a617384ee3aa88bc78243d7949, State: Initialized, Role: FOLLOWER
I20260924 07:10:09.765079 8420 consensus_queue.cc:260] T e25ba7ddc62843ee9409a52a1ac4e23e P 5f1865a617384ee3aa88bc78243d7949 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.765164 8420 ts_tablet_manager.cc:1428] T e25ba7ddc62843ee9409a52a1ac4e23e P 5f1865a617384ee3aa88bc78243d7949: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:09.918140 8473 raft_consensus.cc:482] T e25ba7ddc62843ee9409a52a1ac4e23e P 83906483277942bab937e411fc4c666e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:10:09.918229 8473 raft_consensus.cc:504] T e25ba7ddc62843ee9409a52a1ac4e23e P 83906483277942bab937e411fc4c666e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.918385 8473 leader_election.cc:288] T e25ba7ddc62843ee9409a52a1ac4e23e P 83906483277942bab937e411fc4c666e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f1865a617384ee3aa88bc78243d7949 (127.0.98.2:42777), 3ba1e1457a494fc58f23c23c7e156a59 (127.0.98.3:44545)
I20260924 07:10:09.918654 8337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e25ba7ddc62843ee9409a52a1ac4e23e" candidate_uuid: "83906483277942bab937e411fc4c666e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ba1e1457a494fc58f23c23c7e156a59" is_pre_election: true
I20260924 07:10:09.918707 8202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e25ba7ddc62843ee9409a52a1ac4e23e" candidate_uuid: "83906483277942bab937e411fc4c666e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1865a617384ee3aa88bc78243d7949" is_pre_election: true
I20260924 07:10:09.918792 8337 raft_consensus.cc:2454] T e25ba7ddc62843ee9409a52a1ac4e23e P 3ba1e1457a494fc58f23c23c7e156a59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83906483277942bab937e411fc4c666e in term 0.
I20260924 07:10:09.918816 8202 raft_consensus.cc:2454] T e25ba7ddc62843ee9409a52a1ac4e23e P 5f1865a617384ee3aa88bc78243d7949 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83906483277942bab937e411fc4c666e in term 0.
I20260924 07:10:09.918942 7993 leader_election.cc:302] T e25ba7ddc62843ee9409a52a1ac4e23e P 83906483277942bab937e411fc4c666e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ba1e1457a494fc58f23c23c7e156a59, 83906483277942bab937e411fc4c666e; no voters:
I20260924 07:10:09.919052 8473 raft_consensus.cc:2781] T e25ba7ddc62843ee9409a52a1ac4e23e P 83906483277942bab937e411fc4c666e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:10:09.919108 8473 raft_consensus.cc:482] T e25ba7ddc62843ee9409a52a1ac4e23e P 83906483277942bab937e411fc4c666e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:10:09.919123 8473 raft_consensus.cc:3037] T e25ba7ddc62843ee9409a52a1ac4e23e P 83906483277942bab937e411fc4c666e [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:09.919770 8473 raft_consensus.cc:504] T e25ba7ddc62843ee9409a52a1ac4e23e P 83906483277942bab937e411fc4c666e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.919902 8473 leader_election.cc:288] T e25ba7ddc62843ee9409a52a1ac4e23e P 83906483277942bab937e411fc4c666e [CANDIDATE]: Term 1 election: Requested vote from peers 5f1865a617384ee3aa88bc78243d7949 (127.0.98.2:42777), 3ba1e1457a494fc58f23c23c7e156a59 (127.0.98.3:44545)
I20260924 07:10:09.920032 8202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e25ba7ddc62843ee9409a52a1ac4e23e" candidate_uuid: "83906483277942bab937e411fc4c666e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f1865a617384ee3aa88bc78243d7949"
I20260924 07:10:09.920111 8202 raft_consensus.cc:3037] T e25ba7ddc62843ee9409a52a1ac4e23e P 5f1865a617384ee3aa88bc78243d7949 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:09.920096 8337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e25ba7ddc62843ee9409a52a1ac4e23e" candidate_uuid: "83906483277942bab937e411fc4c666e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ba1e1457a494fc58f23c23c7e156a59"
I20260924 07:10:09.920163 8337 raft_consensus.cc:3037] T e25ba7ddc62843ee9409a52a1ac4e23e P 3ba1e1457a494fc58f23c23c7e156a59 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:09.920852 8202 raft_consensus.cc:2454] T e25ba7ddc62843ee9409a52a1ac4e23e P 5f1865a617384ee3aa88bc78243d7949 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83906483277942bab937e411fc4c666e in term 1.
I20260924 07:10:09.920984 8337 raft_consensus.cc:2454] T e25ba7ddc62843ee9409a52a1ac4e23e P 3ba1e1457a494fc58f23c23c7e156a59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83906483277942bab937e411fc4c666e in term 1.
I20260924 07:10:09.921033 7994 leader_election.cc:302] T e25ba7ddc62843ee9409a52a1ac4e23e P 83906483277942bab937e411fc4c666e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f1865a617384ee3aa88bc78243d7949, 83906483277942bab937e411fc4c666e; no voters:
I20260924 07:10:09.921130 8473 raft_consensus.cc:2781] T e25ba7ddc62843ee9409a52a1ac4e23e P 83906483277942bab937e411fc4c666e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:10:09.921190 8473 raft_consensus.cc:686] T e25ba7ddc62843ee9409a52a1ac4e23e P 83906483277942bab937e411fc4c666e [term 1 LEADER]: Becoming Leader. State: Replica: 83906483277942bab937e411fc4c666e, State: Running, Role: LEADER
I20260924 07:10:09.921252 8473 consensus_queue.cc:237] T e25ba7ddc62843ee9409a52a1ac4e23e P 83906483277942bab937e411fc4c666e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } }
I20260924 07:10:09.921875 7905 catalog_manager.cc:5495] T e25ba7ddc62843ee9409a52a1ac4e23e P 83906483277942bab937e411fc4c666e reported cstate change: term changed from 0 to 1, leader changed from <none> to 83906483277942bab937e411fc4c666e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "83906483277942bab937e411fc4c666e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83906483277942bab937e411fc4c666e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 } health_report { overall_health: UNKNOWN } } }
I20260924 07:10:09.953689 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:09.949569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43660 (local address 127.0.98.1:45771)
0924 07:10:09.950596 (+ 1027us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:09.950598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:09.950607 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:10:09.950631 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:09.950632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:09.950642 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:09.950707 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:10:09.950902 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.951628 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.952773 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:09.953126 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:09.953197 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:09.953376 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:10:09.953427 (+ 51us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:10:09.953472 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:09.953568 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 07:10:09.953590 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1003}
I20260924 07:10:09.956525 8337 raft_consensus.cc:1261] T e25ba7ddc62843ee9409a52a1ac4e23e P 3ba1e1457a494fc58f23c23c7e156a59 [term 1 FOLLOWER]: Refusing update from remote peer 83906483277942bab937e411fc4c666e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:10:09.956550 8202 raft_consensus.cc:1261] T e25ba7ddc62843ee9409a52a1ac4e23e P 5f1865a617384ee3aa88bc78243d7949 [term 1 FOLLOWER]: Refusing update from remote peer 83906483277942bab937e411fc4c666e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:10:09.956763 8473 consensus_queue.cc:1035] T e25ba7ddc62843ee9409a52a1ac4e23e P 83906483277942bab937e411fc4c666e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba1e1457a494fc58f23c23c7e156a59" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:10:09.956863 8472 consensus_queue.cc:1035] T e25ba7ddc62843ee9409a52a1ac4e23e P 83906483277942bab937e411fc4c666e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f1865a617384ee3aa88bc78243d7949" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:10:09.966176 7904 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:10:09.966270 7904 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:10:09.966862 7904 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32fa0114c29249baacf9a200be709192: Sending DeleteTablet for 3 replicas of tablet e25ba7ddc62843ee9409a52a1ac4e23e
I20260924 07:10:09.967100 8322 tablet_service.cc:1514] Processing DeleteTablet for tablet e25ba7ddc62843ee9409a52a1ac4e23e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:10:09 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39354
I20260924 07:10:09.967128 8182 tablet_service.cc:1514] Processing DeleteTablet for tablet e25ba7ddc62843ee9409a52a1ac4e23e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:10:09 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47584
I20260924 07:10:09.967134 8042 tablet_service.cc:1514] Processing DeleteTablet for tablet e25ba7ddc62843ee9409a52a1ac4e23e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:10:09 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43628
I20260924 07:10:09.967196 8438 tablet_replica.cc:330] T e25ba7ddc62843ee9409a52a1ac4e23e P 3ba1e1457a494fc58f23c23c7e156a59: stopping tablet replica
I20260924 07:10:09.967218 8440 tablet_replica.cc:330] T e25ba7ddc62843ee9409a52a1ac4e23e P 83906483277942bab937e411fc4c666e: stopping tablet replica
I20260924 07:10:09.967226 8439 tablet_replica.cc:330] T e25ba7ddc62843ee9409a52a1ac4e23e P 5f1865a617384ee3aa88bc78243d7949: stopping tablet replica
I20260924 07:10:09.967242 8438 raft_consensus.cc:2229] T e25ba7ddc62843ee9409a52a1ac4e23e P 3ba1e1457a494fc58f23c23c7e156a59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:10:09.967262 8440 raft_consensus.cc:2229] T e25ba7ddc62843ee9409a52a1ac4e23e P 83906483277942bab937e411fc4c666e [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:10:09.967276 8438 raft_consensus.cc:2258] T e25ba7ddc62843ee9409a52a1ac4e23e P 3ba1e1457a494fc58f23c23c7e156a59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:10:09.967271 8439 raft_consensus.cc:2229] T e25ba7ddc62843ee9409a52a1ac4e23e P 5f1865a617384ee3aa88bc78243d7949 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:10:09.967301 8439 raft_consensus.cc:2258] T e25ba7ddc62843ee9409a52a1ac4e23e P 5f1865a617384ee3aa88bc78243d7949 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:10:09.967324 8440 raft_consensus.cc:2258] T e25ba7ddc62843ee9409a52a1ac4e23e P 83906483277942bab937e411fc4c666e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:10:09.967628 8439 ts_tablet_manager.cc:1905] T e25ba7ddc62843ee9409a52a1ac4e23e P 5f1865a617384ee3aa88bc78243d7949: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:10:09.967662 8438 ts_tablet_manager.cc:1905] T e25ba7ddc62843ee9409a52a1ac4e23e P 3ba1e1457a494fc58f23c23c7e156a59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:10:09.967747 8440 ts_tablet_manager.cc:1905] T e25ba7ddc62843ee9409a52a1ac4e23e P 83906483277942bab937e411fc4c666e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:10:09.969225 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 7977
I20260924 07:10:09.969885 8438 ts_tablet_manager.cc:1918] T e25ba7ddc62843ee9409a52a1ac4e23e P 3ba1e1457a494fc58f23c23c7e156a59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:10:09.969885 8439 ts_tablet_manager.cc:1918] T e25ba7ddc62843ee9409a52a1ac4e23e P 5f1865a617384ee3aa88bc78243d7949: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 07:10:09.969940 8438 log.cc:1196] T e25ba7ddc62843ee9409a52a1ac4e23e P 3ba1e1457a494fc58f23c23c7e156a59: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-2/wal/wals/e25ba7ddc62843ee9409a52a1ac4e23e
I20260924 07:10:09.969940 8439 log.cc:1196] T e25ba7ddc62843ee9409a52a1ac4e23e P 5f1865a617384ee3aa88bc78243d7949: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790233772407728-392-0/minicluster-data/ts-1/wal/wals/e25ba7ddc62843ee9409a52a1ac4e23e
I20260924 07:10:09.970160 8439 ts_tablet_manager.cc:1939] T e25ba7ddc62843ee9409a52a1ac4e23e P 5f1865a617384ee3aa88bc78243d7949: Deleting consensus metadata
I20260924 07:10:09.970162 8438 ts_tablet_manager.cc:1939] T e25ba7ddc62843ee9409a52a1ac4e23e P 3ba1e1457a494fc58f23c23c7e156a59: Deleting consensus metadata
I20260924 07:10:09.970451 7890 catalog_manager.cc:4841] TS 5f1865a617384ee3aa88bc78243d7949 (127.0.98.2:42777): tablet e25ba7ddc62843ee9409a52a1ac4e23e (table test-table [id=deea0a214942429bbb09c78a0d52a3dd]) successfully deleted
I20260924 07:10:09.970471 7889 catalog_manager.cc:4841] TS 3ba1e1457a494fc58f23c23c7e156a59 (127.0.98.3:44545): tablet e25ba7ddc62843ee9409a52a1ac4e23e (table test-table [id=deea0a214942429bbb09c78a0d52a3dd]) successfully deleted
W20260924 07:10:09.976310 7890 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45771 (error 108)
I20260924 07:10:09.976606 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 8117
I20260924 07:10:09.976856 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:09.976650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45771 (local address 127.0.0.1:43666)
0924 07:10:09.976773 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:09.976811 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":60,"threads_started":1}
W20260924 07:10:09.977814 7890 catalog_manager.cc:4584] TS 83906483277942bab937e411fc4c666e (127.0.98.1:45771): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e25ba7ddc62843ee9409a52a1ac4e23e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45771: connect: Connection refused (error 111)
I20260924 07:10:09.981395 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 8257
I20260924 07:10:09.987215 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 7873
2026-09-24T07:10:09Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2733 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:10:10 dist-test-slave-14ww krb5kdc[8506](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:10:10 dist-test-slave-14ww krb5kdc[8506](info): set up 2 sockets
Sep 24 07:10:10 dist-test-slave-14ww krb5kdc[8506](info): commencing operation
krb5kdc: starting...
W20260924 07:10:11.025327 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:10:11 dist-test-slave-14ww krb5kdc[8506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:10:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:10:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:10:11.139905 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35737
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:11.219520 8519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:11.219709 8519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:11.219730 8519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:11.220892 8519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:10:11.220930 8519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:11.220944 8519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:11.220956 8519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:10:11.221009 8519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:10:11.222453 8519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:10:11.222693 8519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:11.222909 8519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:11.227789 8525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:11.227831 8527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:11.227905 8524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:11.228010 8519 server_base.cc:1034] running on GCE node
I20260924 07:10:11.228253 8519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:11.228574 8519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:11.229733 8519 hybrid_clock.cc:648] HybridClock initialized: now 1790233811229720 us; error 26 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T07:10:11Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1248 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:10:11 dist-test-slave-14ww krb5kdc[8532](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:10:11 dist-test-slave-14ww krb5kdc[8532](info): set up 2 sockets
Sep 24 07:10:11 dist-test-slave-14ww krb5kdc[8532](info): commencing operation
krb5kdc: starting...
W20260924 07:10:12.301285 392 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:10:12 dist-test-slave-14ww krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:10:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:10:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:10:12.419296 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37663
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37663
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:12.497717 8545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:12.497941 8545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:12.497975 8545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:12.499135 8545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:10:12.499202 8545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:12.499236 8545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:12.499285 8545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:10:12.499315 8545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:10:12.500842 8545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:10:12.501101 8545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:12.501336 8545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:12.506325 8553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:12.506421 8550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:12.506330 8551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:12.506486 8545 server_base.cc:1034] running on GCE node
I20260924 07:10:12.506845 8545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:12.507086 8545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:12.508232 8545 hybrid_clock.cc:648] HybridClock initialized: now 1790233812508201 us; error 46 us; skew 500 ppm
Sep 24 07:10:12 dist-test-slave-14ww krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:12.518756 8545 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:10:12.519198 8545 webserver.cc:460] Webserver started at http://127.0.98.62:35975/ using document root <none> and password file <none>
I20260924 07:10:12.519430 8545 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:12.519475 8545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:12.519613 8545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:12.520669 8545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "5e1d13e572494725afae2a6f6c054726"
format_stamp: "Formatted at 2026-09-24 07:10:12 on dist-test-slave-14ww"
server_key: "92c4da62d1d68e03704f02ecd7a6fc1c"
server_key_iv: "8a3c9fc1e15a3a7209e81b82e733230f"
server_key_version: "encryptionkey@0"
I20260924 07:10:12.521113 8545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "5e1d13e572494725afae2a6f6c054726"
format_stamp: "Formatted at 2026-09-24 07:10:12 on dist-test-slave-14ww"
server_key: "92c4da62d1d68e03704f02ecd7a6fc1c"
server_key_iv: "8a3c9fc1e15a3a7209e81b82e733230f"
server_key_version: "encryptionkey@0"
I20260924 07:10:12.522457 8545 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:10:12.523231 8560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:12.523437 8545 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:10:12.523506 8545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "5e1d13e572494725afae2a6f6c054726"
format_stamp: "Formatted at 2026-09-24 07:10:12 on dist-test-slave-14ww"
server_key: "92c4da62d1d68e03704f02ecd7a6fc1c"
server_key_iv: "8a3c9fc1e15a3a7209e81b82e733230f"
server_key_version: "encryptionkey@0"
I20260924 07:10:12.523594 8545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:12.533066 8545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:12.534771 8545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:12.534906 8545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:12.538885 8545 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37663
I20260924 07:10:12.538914 8612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37663 every 8 connection(s)
I20260924 07:10:12.539389 8545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:10:12.539992 8613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:12.542152 8613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726: Bootstrap starting.
I20260924 07:10:12.542670 8613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:12.542913 8613 log.cc:824] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726: Log is configured to *not* fsync() on all Append() calls
I20260924 07:10:12.543290 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 8545
I20260924 07:10:12.543406 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:10:12.543605 392 external_mini_cluster.cc:1428] Setting key b8eef048fbfca4295a6528c6fd8cd636
I20260924 07:10:12.543617 8613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726: No bootstrap required, opened a new log
I20260924 07:10:12.545959 8613 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e1d13e572494725afae2a6f6c054726" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37663 } }
I20260924 07:10:12.546087 8613 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:12.546108 8613 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1d13e572494725afae2a6f6c054726, State: Initialized, Role: FOLLOWER
I20260924 07:10:12.546262 8613 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e1d13e572494725afae2a6f6c054726" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37663 } }
I20260924 07:10:12.546362 8613 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:10:12.546398 8613 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:10:12.546450 8613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:12.547134 8613 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e1d13e572494725afae2a6f6c054726" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37663 } }
I20260924 07:10:12.547219 8613 leader_election.cc:302] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e1d13e572494725afae2a6f6c054726; no voters:
I20260924 07:10:12.547394 8613 leader_election.cc:288] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:10:12.547456 8618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:10:12.547652 8613 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:10:12.547655 8618 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726 [term 1 LEADER]: Becoming Leader. State: Replica: 5e1d13e572494725afae2a6f6c054726, State: Running, Role: LEADER
I20260924 07:10:12.547902 8618 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e1d13e572494725afae2a6f6c054726" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37663 } }
I20260924 07:10:12.548277 8619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e1d13e572494725afae2a6f6c054726" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e1d13e572494725afae2a6f6c054726" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37663 } } }
I20260924 07:10:12.548959 8619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726 [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:12.548255 8620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e1d13e572494725afae2a6f6c054726. Latest consensus state: current_term: 1 leader_uuid: "5e1d13e572494725afae2a6f6c054726" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e1d13e572494725afae2a6f6c054726" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37663 } } }
I20260924 07:10:12.549196 8620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726 [sys.catalog]: This master's current role is: LEADER
W20260924 07:10:12.549590 8634 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:10:12.549657 8634 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:10:12.549732 8633 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:10:12.549929 8633 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:10:12.551298 8633 catalog_manager.cc:1347] Generated new cluster ID: 1717da8eb9744c1db44e2acd8bb7f3cf
I20260924 07:10:12.551366 8633 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 07:10:12 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:10:12.556703 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:12.544097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36118 (local address 127.0.98.62:37663)
0924 07:10:12.544260 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:12.544268 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:12.544292 (+ 24us) server_negotiation.cc:407] Connection header received
0924 07:10:12.545955 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:12.545963 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:12.546022 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:12.546195 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:12.546971 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:12.548335 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:12.549516 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:12.550103 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:12.550204 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:12.553166 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:12.553170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:12.553170 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:12.553181 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:12.555086 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:12.555516 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:12.555519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:12.555519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:12.555631 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:12.555850 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:12.555851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:12.555851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:12.556126 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:12.556281 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:12.556387 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 07:10:12.556510 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
I20260924 07:10:12.561939 8633 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:10:12.562592 8633 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:10:12.566133 8633 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726: Generated new TSK 0
I20260924 07:10:12.566286 8633 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 07:10:12.596251 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37663
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:12.676991 8639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:12.677194 8639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:12.677229 8639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:12.678387 8639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:12.678452 8639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:12.678495 8639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:10:12.679967 8639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:10:12.680215 8639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:12.680426 8639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:12.685235 8645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:12.685274 8644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:12.685249 8647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:12.685482 8639 server_base.cc:1034] running on GCE node
I20260924 07:10:12.685675 8639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:12.685941 8639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:12.687083 8639 hybrid_clock.cc:648] HybridClock initialized: now 1790233812687073 us; error 26 us; skew 500 ppm
Sep 24 07:10:12 dist-test-slave-14ww krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:12.689577 8639 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:10:12.689999 8639 webserver.cc:460] Webserver started at http://127.0.98.1:40377/ using document root <none> and password file <none>
I20260924 07:10:12.690207 8639 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:12.690271 8639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:12.690411 8639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:12.691377 8639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-0/data/instance:
uuid: "66bbf657286443f89d9ff4829ff93335"
format_stamp: "Formatted at 2026-09-24 07:10:12 on dist-test-slave-14ww"
server_key: "2d6853d7c252aa8e4bfd93ec6c858910"
server_key_iv: "acb51e8532872c0215eea58863870429"
server_key_version: "encryptionkey@0"
I20260924 07:10:12.691715 8639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-0/wal/instance:
uuid: "66bbf657286443f89d9ff4829ff93335"
format_stamp: "Formatted at 2026-09-24 07:10:12 on dist-test-slave-14ww"
server_key: "2d6853d7c252aa8e4bfd93ec6c858910"
server_key_iv: "acb51e8532872c0215eea58863870429"
server_key_version: "encryptionkey@0"
I20260924 07:10:12.693004 8639 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:10:12.693697 8654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:12.693868 8639 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:10:12.693959 8639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-0/wal
uuid: "66bbf657286443f89d9ff4829ff93335"
format_stamp: "Formatted at 2026-09-24 07:10:12 on dist-test-slave-14ww"
server_key: "2d6853d7c252aa8e4bfd93ec6c858910"
server_key_iv: "acb51e8532872c0215eea58863870429"
server_key_version: "encryptionkey@0"
I20260924 07:10:12.694031 8639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:12.700153 8639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:12.702311 8639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:12.702464 8639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:12.702672 8639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:10:12.702943 8639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:10:12.702994 8639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:12.703032 8639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:10:12.703058 8639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:12.708560 8639 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40683
I20260924 07:10:12.708629 8767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40683 every 8 connection(s)
I20260924 07:10:12.708952 8639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:10:12.711208 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 8639
I20260924 07:10:12.711308 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:10:12.711498 392 external_mini_cluster.cc:1428] Setting key 074279fde87880a461d7b9c646afa33a
Sep 24 07:10:12 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:10:12.719110 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:12.709584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41903 (local address 127.0.98.62:37663)
0924 07:10:12.709623 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:12.709625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:12.709818 (+ 193us) server_negotiation.cc:407] Connection header received
0924 07:10:12.710675 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:12.710676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:12.710689 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:12.710744 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:12.711356 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:12.712079 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:12.713659 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:12.713996 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:12.714069 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:12.716037 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:12.716039 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:12.716039 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:12.716049 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:12.717758 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:12.718123 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:12.718125 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:12.718125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:12.718197 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:12.718419 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:12.718420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:12.718421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:12.718566 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:12.718655 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:12.718920 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 07:10:12.719015 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 07:10:12.719123 8770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:12.709578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.1:41903)
0924 07:10:12.709707 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:12.709738 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:12.710592 (+ 854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:12.710794 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:12.710797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:12.711026 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:12.711219 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:12.711222 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:12.712172 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:12.712173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:12.713555 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:12.713559 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:12.714078 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:12.714078 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:12.714169 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:12.714652 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:12.714660 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:12.715944 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:12.717848 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:12.717853 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:12.717855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:12.718050 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:12.718268 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:12.718268 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:12.718268 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:12.718353 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:12.718649 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:12.718653 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:12.718789 (+ 136us) client_negotiation.cc:769] Sending connection context
0924 07:10:12.718869 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 07:10:12.718977 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":72,"threads_started":1}
I20260924 07:10:12.719851 8768 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37663
I20260924 07:10:12.719944 8768 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 07:10:12.720114 8768 heartbeater.cc:511] Master 127.0.98.62:37663 requested a full tablet report, sending...
I20260924 07:10:12.720571 8577 ts_manager.cc:194] Registered new tserver with Master: 66bbf657286443f89d9ff4829ff93335 (127.0.98.1:40683)
I20260924 07:10:12.721216 8577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41903
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 07:10:12.741647 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37663
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:12.830947 8773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:12.831204 8773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:12.831243 8773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:12.833381 8773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:12.833457 8773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:12.833501 8773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:10:12.835568 8773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:10:12.835824 8773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:12.836056 8773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:12.841097 8781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:12.841090 8779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:12.841257 8773 server_base.cc:1034] running on GCE node
W20260924 07:10:12.841104 8778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:12.841648 8773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:12.841928 8773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:12.843106 8773 hybrid_clock.cc:648] HybridClock initialized: now 1790233812843083 us; error 38 us; skew 500 ppm
Sep 24 07:10:12 dist-test-slave-14ww krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:12.846417 8773 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:10:12.846946 8773 webserver.cc:460] Webserver started at http://127.0.98.2:39827/ using document root <none> and password file <none>
I20260924 07:10:12.847237 8773 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:12.847309 8773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:12.847451 8773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:12.848605 8773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-1/data/instance:
uuid: "8c19bb870a05492080a9085dffb2b5a8"
format_stamp: "Formatted at 2026-09-24 07:10:12 on dist-test-slave-14ww"
server_key: "edd7b45018a02c1a77d9e76ef7505147"
server_key_iv: "9233160401c8510176aa7c7632c1a718"
server_key_version: "encryptionkey@0"
I20260924 07:10:12.848963 8773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8c19bb870a05492080a9085dffb2b5a8"
format_stamp: "Formatted at 2026-09-24 07:10:12 on dist-test-slave-14ww"
server_key: "edd7b45018a02c1a77d9e76ef7505147"
server_key_iv: "9233160401c8510176aa7c7632c1a718"
server_key_version: "encryptionkey@0"
I20260924 07:10:12.850275 8773 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:10:12.850921 8788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:12.851101 8773 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:10:12.851197 8773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-1/wal
uuid: "8c19bb870a05492080a9085dffb2b5a8"
format_stamp: "Formatted at 2026-09-24 07:10:12 on dist-test-slave-14ww"
server_key: "edd7b45018a02c1a77d9e76ef7505147"
server_key_iv: "9233160401c8510176aa7c7632c1a718"
server_key_version: "encryptionkey@0"
I20260924 07:10:12.851266 8773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:12.862296 8773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:12.864125 8773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:12.864272 8773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:12.864491 8773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:10:12.864876 8773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:10:12.864914 8773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:12.864964 8773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:10:12.864992 8773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:12.870718 8773 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44463
I20260924 07:10:12.870796 8901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44463 every 8 connection(s)
I20260924 07:10:12.871097 8773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:10:12.875939 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 8773
I20260924 07:10:12.876076 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:10:12.876307 392 external_mini_cluster.cc:1428] Setting key c7fd9e7a328a06305df3cd44dd7a7b6d
Sep 24 07:10:12 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:10:12.881378 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:12.871789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59149 (local address 127.0.98.62:37663)
0924 07:10:12.871864 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:12.871866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:12.872083 (+ 217us) server_negotiation.cc:407] Connection header received
0924 07:10:12.873169 (+ 1086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:12.873171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:12.873182 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:12.873236 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:12.873796 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:12.874287 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:12.875357 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:12.875561 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:12.875626 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:12.878266 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:12.878267 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:12.878268 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:12.878275 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:12.879927 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:12.880313 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:12.880314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:12.880315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:12.880379 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:12.880607 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:12.880608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:12.880609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:12.880797 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:12.880876 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:12.881173 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 07:10:12.881263 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 07:10:12.881454 8904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:12.871786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.2:59149)
0924 07:10:12.871948 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:12.871989 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:12.873051 (+ 1062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:12.873298 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:12.873302 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:12.873536 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:12.873723 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:12.873726 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:12.874389 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:12.874390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:12.875244 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:12.875247 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:12.875656 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:12.875657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:12.875813 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:12.876388 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:12.876398 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:12.878169 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:12.880008 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:12.880013 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:12.880015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:12.880237 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:12.880447 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:12.880447 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:12.880448 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:12.880540 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:12.880878 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:12.880885 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:12.881038 (+ 153us) client_negotiation.cc:769] Sending connection context
0924 07:10:12.881124 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 07:10:12.881263 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":100,"threads_started":1}
I20260924 07:10:12.881947 8902 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37663
I20260924 07:10:12.882061 8902 heartbeater.cc:464] Registering TS with master...
I20260924 07:10:12.882233 8902 heartbeater.cc:511] Master 127.0.98.62:37663 requested a full tablet report, sending...
I20260924 07:10:12.882684 8577 ts_manager.cc:194] Registered new tserver with Master: 8c19bb870a05492080a9085dffb2b5a8 (127.0.98.2:44463)
I20260924 07:10:12.883226 8577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59149
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 07:10:12.909257 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37663
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:12.989307 8907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:12.989504 8907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:12.989527 8907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:12.990593 8907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:12.990626 8907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:12.990651 8907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:10:12.992041 8907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:10:12.992226 8907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:12.992393 8907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:12.997257 8913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:12.997293 8907 server_base.cc:1034] running on GCE node
W20260924 07:10:12.997273 8915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:12.997267 8912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:12.997658 8907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:12.997843 8907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:12.999035 8907 hybrid_clock.cc:648] HybridClock initialized: now 1790233812999031 us; error 37 us; skew 500 ppm
Sep 24 07:10:12 dist-test-slave-14ww krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:13.001643 8907 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:10:13.002025 8907 webserver.cc:460] Webserver started at http://127.0.98.3:41281/ using document root <none> and password file <none>
I20260924 07:10:13.002225 8907 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:13.002265 8907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:13.002350 8907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:13.003333 8907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-2/data/instance:
uuid: "ef669ed6906b4394ac3908475887cb3f"
format_stamp: "Formatted at 2026-09-24 07:10:12 on dist-test-slave-14ww"
server_key: "0f73c32f0aa87d0424df9c3b5fabbc4e"
server_key_iv: "d6fdd73549fb03af0006144063689fff"
server_key_version: "encryptionkey@0"
I20260924 07:10:13.003630 8907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ef669ed6906b4394ac3908475887cb3f"
format_stamp: "Formatted at 2026-09-24 07:10:12 on dist-test-slave-14ww"
server_key: "0f73c32f0aa87d0424df9c3b5fabbc4e"
server_key_iv: "d6fdd73549fb03af0006144063689fff"
server_key_version: "encryptionkey@0"
I20260924 07:10:13.004892 8907 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:10:13.005649 8922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:13.005959 8907 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:10:13.006031 8907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-2/wal
uuid: "ef669ed6906b4394ac3908475887cb3f"
format_stamp: "Formatted at 2026-09-24 07:10:12 on dist-test-slave-14ww"
server_key: "0f73c32f0aa87d0424df9c3b5fabbc4e"
server_key_iv: "d6fdd73549fb03af0006144063689fff"
server_key_version: "encryptionkey@0"
I20260924 07:10:13.006125 8907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:13.016609 8907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:13.018707 8907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:13.018817 8907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:13.018996 8907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:10:13.019312 8907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:10:13.019368 8907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:13.019415 8907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:10:13.019440 8907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:13.025254 8907 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43715
I20260924 07:10:13.025367 9035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43715 every 8 connection(s)
I20260924 07:10:13.025648 8907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 07:10:13 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:10:13.033677 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 8907
I20260924 07:10:13.033820 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:10:13.034018 392 external_mini_cluster.cc:1428] Setting key 2559e9052082572e0ef5b61175819664
I20260924 07:10:13.035303 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:13.026441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33509 (local address 127.0.98.62:37663)
0924 07:10:13.026513 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:13.026516 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:13.026707 (+ 191us) server_negotiation.cc:407] Connection header received
0924 07:10:13.027833 (+ 1126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:13.027834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:13.027845 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:13.027895 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:13.028462 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.029007 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.029962 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.030143 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.030196 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:13.032199 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:13.032201 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:13.032201 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:13.032208 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:13.033884 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:13.034380 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:13.034382 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:13.034382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:13.034453 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:13.034740 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:13.034742 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:13.034742 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:13.034876 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:13.034950 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:13.035163 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 07:10:13.035228 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 07:10:13.035476 9038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:13.026403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.3:33509)
0924 07:10:13.026586 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:13.026623 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:13.027746 (+ 1123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:13.027957 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:13.027960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:13.028210 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:13.028395 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:13.028399 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.029082 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.029082 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:13.029884 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:13.029886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.030216 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.030216 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:13.030309 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:13.030819 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:13.030828 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:13.032117 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:13.033985 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:13.033991 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:13.033993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:13.034286 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:13.034531 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:13.034531 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:13.034532 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:13.034671 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:13.034946 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:13.034952 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:13.035065 (+ 113us) client_negotiation.cc:769] Sending connection context
0924 07:10:13.035148 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 07:10:13.035258 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":92,"threads_started":1}
I20260924 07:10:13.035944 9036 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37663
I20260924 07:10:13.036041 9036 heartbeater.cc:464] Registering TS with master...
I20260924 07:10:13.036211 9036 heartbeater.cc:511] Master 127.0.98.62:37663 requested a full tablet report, sending...
I20260924 07:10:13.036559 8577 ts_manager.cc:194] Registered new tserver with Master: ef669ed6906b4394ac3908475887cb3f (127.0.98.3:43715)
I20260924 07:10:13.037003 8577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33509
I20260924 07:10:13.045715 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 07:10:13.156662 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 8639
I20260924 07:10:13.161245 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 8773
I20260924 07:10:13.165794 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 8907
I20260924 07:10:13.170204 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 8545
I20260924 07:10:13.175268 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37663
--webserver_interface=127.0.98.62
--webserver_port=35975
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37663
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:13.257140 9045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:13.257375 9045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:13.257412 9045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:13.258594 9045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:10:13.258668 9045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:13.258695 9045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:13.258728 9045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:10:13.258754 9045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:10:13.260214 9045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:10:13.260484 9045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:13.260715 9045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:13.265522 9053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:13.265647 9051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:13.265699 9050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:13.265683 9045 server_base.cc:1034] running on GCE node
I20260924 07:10:13.266139 9045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:13.266356 9045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:13.267504 9045 hybrid_clock.cc:648] HybridClock initialized: now 1790233813267491 us; error 27 us; skew 500 ppm
Sep 24 07:10:13 dist-test-slave-14ww krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:13.270201 9045 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:10:13.270645 9045 webserver.cc:460] Webserver started at http://127.0.98.62:35975/ using document root <none> and password file <none>
I20260924 07:10:13.270875 9045 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:13.270931 9045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:13.272294 9045 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:10:13.273118 9060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:13.273386 9045 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:10:13.273500 9045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "5e1d13e572494725afae2a6f6c054726"
format_stamp: "Formatted at 2026-09-24 07:10:12 on dist-test-slave-14ww"
server_key: "92c4da62d1d68e03704f02ecd7a6fc1c"
server_key_iv: "8a3c9fc1e15a3a7209e81b82e733230f"
server_key_version: "encryptionkey@0"
I20260924 07:10:13.273849 9045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:13.301004 9045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:13.303397 9045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:13.303560 9045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:13.307507 9045 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37663
I20260924 07:10:13.307548 9112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37663 every 8 connection(s)
I20260924 07:10:13.308151 9045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:10:13.309471 9113 sys_catalog.cc:263] Verifying existing consensus state
I20260924 07:10:13.309923 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 9045
I20260924 07:10:13.310086 9113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726: Bootstrap starting.
I20260924 07:10:13.310307 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40683
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37663
--webserver_port=40377
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:10:13.318526 9113 log.cc:824] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726: Log is configured to *not* fsync() on all Append() calls
I20260924 07:10:13.320207 9113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:10:13.320504 9113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726: Bootstrap complete.
I20260924 07:10:13.322803 9113 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e1d13e572494725afae2a6f6c054726" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37663 } }
I20260924 07:10:13.323019 9113 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1d13e572494725afae2a6f6c054726, State: Initialized, Role: FOLLOWER
I20260924 07:10:13.323202 9113 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e1d13e572494725afae2a6f6c054726" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37663 } }
I20260924 07:10:13.323272 9113 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:10:13.323323 9113 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:10:13.323369 9113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726 [term 1 FOLLOWER]: Advancing to term 2
I20260924 07:10:13.324325 9113 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e1d13e572494725afae2a6f6c054726" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37663 } }
I20260924 07:10:13.324409 9113 leader_election.cc:302] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e1d13e572494725afae2a6f6c054726; no voters:
I20260924 07:10:13.324620 9113 leader_election.cc:288] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 07:10:13.324656 9117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 07:10:13.324836 9117 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726 [term 2 LEADER]: Becoming Leader. State: Replica: 5e1d13e572494725afae2a6f6c054726, State: Running, Role: LEADER
I20260924 07:10:13.324911 9113 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:10:13.324986 9117 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e1d13e572494725afae2a6f6c054726" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37663 } }
I20260924 07:10:13.325333 9117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e1d13e572494725afae2a6f6c054726. Latest consensus state: current_term: 2 leader_uuid: "5e1d13e572494725afae2a6f6c054726" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e1d13e572494725afae2a6f6c054726" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37663 } } }
I20260924 07:10:13.325330 9119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5e1d13e572494725afae2a6f6c054726" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e1d13e572494725afae2a6f6c054726" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37663 } } }
I20260924 07:10:13.325527 9117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726 [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:13.325539 9119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726 [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:13.325759 9126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:10:13.326313 9126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:10:13.326486 9126 catalog_manager.cc:1259] Loaded cluster ID: 1717da8eb9744c1db44e2acd8bb7f3cf
I20260924 07:10:13.326710 9126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:10:13.327991 9126 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:10:13.328157 9126 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5e1d13e572494725afae2a6f6c054726: Loaded TSK: 0
I20260924 07:10:13.328352 9126 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 07:10:13.392887 9115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:13.393076 9115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:13.393100 9115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:13.394258 9115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:13.394297 9115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:13.394325 9115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:10:13.395789 9115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40683
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40377
--tserver_master_addrs=127.0.98.62:37663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:10:13.396008 9115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:13.396256 9115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:10:13.401470 9115 server_base.cc:1034] running on GCE node
W20260924 07:10:13.401597 9142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:13.401501 9139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:13.401481 9140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:13.402009 9115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:13.402266 9115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:13.403438 9115 hybrid_clock.cc:648] HybridClock initialized: now 1790233813403417 us; error 37 us; skew 500 ppm
Sep 24 07:10:13 dist-test-slave-14ww krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233813, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:13.406025 9115 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 07:10:13.406538 9115 webserver.cc:460] Webserver started at http://127.0.98.1:40377/ using document root <none> and password file <none>
I20260924 07:10:13.406769 9115 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:13.406831 9115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:13.408213 9115 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:10:13.408879 9149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:13.409092 9115 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 07:10:13.409190 9115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-0/wal
uuid: "66bbf657286443f89d9ff4829ff93335"
format_stamp: "Formatted at 2026-09-24 07:10:12 on dist-test-slave-14ww"
server_key: "2d6853d7c252aa8e4bfd93ec6c858910"
server_key_iv: "acb51e8532872c0215eea58863870429"
server_key_version: "encryptionkey@0"
I20260924 07:10:13.409593 9115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:13.437656 9115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:13.440023 9115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:13.440189 9115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:13.440444 9115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:10:13.440790 9115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:10:13.440855 9115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:13.440899 9115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:10:13.440928 9115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:13.447067 9115 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40683
I20260924 07:10:13.447109 9262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40683 every 8 connection(s)
I20260924 07:10:13.447439 9115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 07:10:13 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:13 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:13.455273 9265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:13.448088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.1:50201)
0924 07:10:13.448290 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:13.448338 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:13.448931 (+ 593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:13.449328 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:13.449332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:13.450026 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:13.450279 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:13.450284 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.451290 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.451291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:13.452069 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:13.452070 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.452660 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.452661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:13.452785 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:13.453185 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:13.453193 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:13.455145 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":110,"threads_started":1}
W20260924 07:10:13.455426 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:13.448205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50201 (local address 127.0.98.62:37663)
0924 07:10:13.448852 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:13.448860 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:13.448888 (+ 28us) server_negotiation.cc:407] Connection header received
0924 07:10:13.449100 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:13.449105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:13.449145 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:13.449250 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:13.450381 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.451202 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.452149 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.452551 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.452639 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:13.455257 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50201: BlockingRecv error: recv got EOF from 127.0.98.1:50201 (error 108)
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":74,"threads_started":1}
I20260924 07:10:13.455622 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 9115
I20260924 07:10:13.455873 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44463
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37663
--webserver_port=39827
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:13.455897 9263 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37663 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:37663: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 24 07:10:13 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:13 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:13.460614 9265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:13.456165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.1:49289)
0924 07:10:13.456227 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:13.456238 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:13.456281 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:13.456421 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:13.456422 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:13.456521 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:13.456559 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:13.456560 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.457467 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.457469 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:13.457973 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:13.457975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.458327 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.458328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:13.458392 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:13.458798 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:13.458803 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:13.460522 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 07:10:13.460641 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:13.456230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49289 (local address 127.0.98.62:37663)
0924 07:10:13.456294 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:13.456296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:13.456331 (+ 35us) server_negotiation.cc:407] Connection header received
0924 07:10:13.456351 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:13.456352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:13.456364 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:13.456421 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:13.456623 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.457382 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.458068 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.458255 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.458334 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:13.460602 (+ 2268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49289: BlockingRecv error: recv got EOF from 127.0.98.1:49289 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 24 07:10:13 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:13 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:13.465687 9265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:13.460927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.1:46339)
0924 07:10:13.461009 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:13.461020 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:13.461064 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:13.461183 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:13.461184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:13.461343 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:13.461378 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:13.461379 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.462038 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.462039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:13.462933 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:13.462936 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.463266 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.463267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:13.463339 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:13.463671 (+ 332us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:13.463676 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:13.465614 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":66}
W20260924 07:10:13.465731 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:13.461008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46339 (local address 127.0.98.62:37663)
0924 07:10:13.461059 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:13.461060 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:13.461069 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:10:13.461125 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:13.461125 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:13.461133 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:13.461176 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:13.461446 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.461951 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.463020 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.463192 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.463254 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:13.465675 (+ 2421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46339: BlockingRecv error: recv got EOF from 127.0.98.1:46339 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260924 07:10:13.466041 9263 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 07:10:13.536599 9267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:13.536806 9267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:13.536829 9267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:13.537973 9267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:13.538012 9267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:13.538038 9267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:10:13.539456 9267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44463
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39827
--tserver_master_addrs=127.0.98.62:37663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:10:13.539685 9267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:13.539908 9267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:13.544699 9273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:13.544816 9272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:13.544759 9275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:13.544869 9267 server_base.cc:1034] running on GCE node
I20260924 07:10:13.545056 9267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:13.545285 9267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:13.546449 9267 hybrid_clock.cc:648] HybridClock initialized: now 1790233813546436 us; error 27 us; skew 500 ppm
Sep 24 07:10:13 dist-test-slave-14ww krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233813, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:13.548835 9267 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 07:10:13.549263 9267 webserver.cc:460] Webserver started at http://127.0.98.2:39827/ using document root <none> and password file <none>
I20260924 07:10:13.549504 9267 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:13.549587 9267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:13.551015 9267 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:10:13.551702 9282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:13.551957 9267 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:10:13.552057 9267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-1/wal
uuid: "8c19bb870a05492080a9085dffb2b5a8"
format_stamp: "Formatted at 2026-09-24 07:10:12 on dist-test-slave-14ww"
server_key: "edd7b45018a02c1a77d9e76ef7505147"
server_key_iv: "9233160401c8510176aa7c7632c1a718"
server_key_version: "encryptionkey@0"
I20260924 07:10:13.552430 9267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:13.561257 9267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:13.563690 9267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:13.563881 9267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:13.564121 9267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:10:13.564443 9267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:10:13.564499 9267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:13.564539 9267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:10:13.564572 9267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:13.570724 9267 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44463
I20260924 07:10:13.570801 9395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44463 every 8 connection(s)
I20260924 07:10:13.571161 9267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 07:10:13 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:13 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:13.578809 9398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:13.571907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.2:33901)
0924 07:10:13.572130 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:13.572167 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:13.572675 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:13.572960 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:13.572964 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:13.573536 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:13.573723 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:13.573728 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.574440 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.574441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:13.575253 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:13.575256 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.575659 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.575660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:13.575786 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:13.576335 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:13.576344 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:13.578668 (+ 2324us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":84,"threads_started":1}
W20260924 07:10:13.578821 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:13.571955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33901 (local address 127.0.98.62:37663)
0924 07:10:13.572037 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:13.572039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:13.572255 (+ 216us) server_negotiation.cc:407] Connection header received
0924 07:10:13.572817 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:13.572819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:13.572833 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:13.572879 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:13.573825 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.574346 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.575348 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.575556 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.575650 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:13.578775 (+ 3125us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33901: BlockingRecv error: recv got EOF from 127.0.98.2:33901 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260924 07:10:13.579080 9396 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37663 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:37663: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260924 07:10:13.580812 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 9267
I20260924 07:10:13.581110 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43715
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37663
--webserver_port=41281
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:10:13 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:13 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:13.584666 9398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:13.579301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.2:56737)
0924 07:10:13.579352 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:13.579363 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:13.579406 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:13.579563 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:13.579563 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:13.579691 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:13.579736 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:13.579738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.580423 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.580424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:13.581297 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:13.581299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.581843 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.581843 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:13.581914 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:13.582337 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:13.582343 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:13.584562 (+ 2219us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 07:10:13.584693 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:13.579352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56737 (local address 127.0.98.62:37663)
0924 07:10:13.579407 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:13.579408 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:13.579417 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:10:13.579460 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:13.579461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:13.579468 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:13.579509 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:13.579867 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.580328 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.581401 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.581711 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.581751 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:13.584648 (+ 2897us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56737: BlockingRecv error: recv got EOF from 127.0.98.2:56737 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 24 07:10:13 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:13 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:13.590466 9398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:13.585051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.2:49297)
0924 07:10:13.585146 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:13.585156 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:13.585195 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:13.585311 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:13.585312 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:13.585467 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:13.585508 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:13.585510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.586126 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.586127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:13.587033 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:13.587036 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.587423 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.587424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:13.587488 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:13.587863 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:13.587870 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:13.590368 (+ 2498us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":80}
W20260924 07:10:13.590732 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:13.585159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49297 (local address 127.0.98.62:37663)
0924 07:10:13.585212 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:13.585214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:13.585230 (+ 16us) server_negotiation.cc:407] Connection header received
0924 07:10:13.585249 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:13.585250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:13.585257 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:13.585305 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:13.585583 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.586047 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.587124 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.587299 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.587417 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:13.590635 (+ 3218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49297: BlockingRecv error: recv got EOF from 127.0.98.2:49297 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260924 07:10:13.591014 9396 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 07:10:13.664793 9399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:13.665016 9399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:13.665051 9399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:13.666193 9399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:13.666323 9399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:13.666404 9399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:10:13.667832 9399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43715
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41281
--tserver_master_addrs=127.0.98.62:37663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:10:13.668092 9399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:13.668314 9399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:13.673195 9405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:13.673190 9404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:13.673205 9399 server_base.cc:1034] running on GCE node
W20260924 07:10:13.673190 9407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:13.673636 9399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:13.673825 9399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:13.674995 9399 hybrid_clock.cc:648] HybridClock initialized: now 1790233813674947 us; error 73 us; skew 500 ppm
Sep 24 07:10:13 dist-test-slave-14ww krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233813, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:13.677558 9399 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 07:10:13.677938 9399 webserver.cc:460] Webserver started at http://127.0.98.3:41281/ using document root <none> and password file <none>
I20260924 07:10:13.678149 9399 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:13.678225 9399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:13.679494 9399 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:10:13.680045 9414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:13.680197 9399 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:10:13.680276 9399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-2/wal
uuid: "ef669ed6906b4394ac3908475887cb3f"
format_stamp: "Formatted at 2026-09-24 07:10:12 on dist-test-slave-14ww"
server_key: "0f73c32f0aa87d0424df9c3b5fabbc4e"
server_key_iv: "d6fdd73549fb03af0006144063689fff"
server_key_version: "encryptionkey@0"
I20260924 07:10:13.680562 9399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:13.694605 9399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:13.696496 9399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:13.696605 9399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:13.696852 9399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:10:13.697172 9399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:10:13.697211 9399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:13.697254 9399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:10:13.697281 9399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:13.702834 9399 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43715
I20260924 07:10:13.702944 9527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43715 every 8 connection(s)
I20260924 07:10:13.703279 9399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:10:13.706548 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 9399
Sep 24 07:10:13 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:13 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 07:10:13.711259 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:13.704032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45697 (local address 127.0.98.62:37663)
0924 07:10:13.704105 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:13.704107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:13.704329 (+ 222us) server_negotiation.cc:407] Connection header received
0924 07:10:13.704853 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:13.704854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:13.704866 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:13.704908 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:13.705889 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.706385 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.707617 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.707812 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.707896 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:13.711208 (+ 3312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45697: BlockingRecv error: recv got EOF from 127.0.98.3:45697 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 07:10:13.711354 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:13.704020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.3:45697)
0924 07:10:13.704205 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:13.704241 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:13.704702 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:13.704993 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:13.704997 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:13.705567 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:13.705779 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:13.705784 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.706481 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.706482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:13.707515 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:13.707518 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.707909 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.707910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:13.708034 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:13.708602 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:13.708610 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:13.711129 (+ 2519us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":96,"threads_started":1}
W20260924 07:10:13.711606 9528 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37663 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:37663: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260924 07:10:13.714792 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:13.707121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36132 (local address 127.0.98.62:37663)
0924 07:10:13.707263 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:13.707265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:13.707273 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:10:13.707308 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:13.707313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:13.707323 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:13.707393 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:13.707992 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.708639 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.709477 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.709791 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.709863 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:13.711413 (+ 1550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:13.711418 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:13.711418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:13.711432 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:13.713443 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:13.713817 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:13.713821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:13.713821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:13.713917 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:13.714118 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:13.714119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:13.714119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:13.714337 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:13.714468 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:13.714597 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 07:10:13.714705 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":71,"threads_started":1}
Sep 24 07:10:13 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:13 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:13.716553 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:13.711829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.3:50605)
0924 07:10:13.711949 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:13.711959 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:13.712013 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:13.712152 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:13.712152 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:13.712268 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:13.712311 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:13.712312 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.713232 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.713232 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:13.713734 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:13.713736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.714122 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.714122 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:13.714187 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:13.714606 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:13.714611 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:13.716476 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105}
W20260924 07:10:13.716598 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:13.711910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50605 (local address 127.0.98.62:37663)
0924 07:10:13.711967 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:13.711969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:13.712002 (+ 33us) server_negotiation.cc:407] Connection header received
0924 07:10:13.712067 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:13.712068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:13.712078 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:13.712143 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:13.712389 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.713140 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.713845 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.714035 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.714120 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:13.716540 (+ 2420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50605: BlockingRecv error: recv got EOF from 127.0.98.3:50605 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 07:10:13 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:13 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:13.720703 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:13.716836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.3:57135)
0924 07:10:13.716895 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:13.716904 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:13.716941 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:13.717055 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:13.717055 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:13.717177 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:13.717209 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:13.717210 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.717850 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.717850 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:13.718333 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:13.718334 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.718710 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.718710 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:13.718760 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:13.719150 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:13.719155 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:13.720631 (+ 1476us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 07:10:13.720777 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:13.716894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57135 (local address 127.0.98.62:37663)
0924 07:10:13.716930 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:13.716932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:13.716940 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:10:13.716997 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:13.716998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:13.717007 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:13.717053 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:13.717309 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.717753 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.718412 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:13.718628 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:13.718699 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:13.720691 (+ 1992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57135: BlockingRecv error: recv got EOF from 127.0.98.3:57135 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 07:10:13.720853 9528 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 07:10:14 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:14 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:14.470959 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:14.466441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.1:45403)
0924 07:10:14.466588 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:14.466598 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:14.466650 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:14.466782 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:14.466783 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:14.466935 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:14.466983 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:14.466984 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:14.467628 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:14.467630 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:14.468298 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:14.468300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:14.468661 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:14.468662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:14.468712 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:14.469230 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:14.469235 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:14.470897 (+ 1662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
W20260924 07:10:14.471062 9534 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:14.466523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45403 (local address 127.0.98.62:37663)
0924 07:10:14.466631 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:14.466633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:14.466642 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:10:14.466695 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:14.466696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:14.466704 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:14.466762 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:14.467048 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:14.467554 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:14.468399 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:14.468590 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:14.468654 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:14.470985 (+ 2331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45403: BlockingRecv error: recv got EOF from 127.0.98.1:45403 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":53,"threads_started":1}
Sep 24 07:10:14 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:14 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:14.595940 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:14.591457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.2:52215)
0924 07:10:14.591600 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:14.591611 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:14.591698 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:14.591821 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:14.591822 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:14.592012 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:14.592059 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:14.592061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:14.592762 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:14.592763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:14.593312 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:14.593314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:14.593707 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:14.593708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:14.593764 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:14.594198 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:14.594203 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:14.595875 (+ 1672us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":61,"threads_started":1}
W20260924 07:10:14.596032 9534 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:14.591483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52215 (local address 127.0.98.62:37663)
0924 07:10:14.591525 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:14.591527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:14.591682 (+ 155us) server_negotiation.cc:407] Connection header received
0924 07:10:14.591752 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:14.591753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:14.591765 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:14.591816 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:14.592140 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:14.592648 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:14.593411 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:14.593619 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:14.593708 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:14.595966 (+ 2258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52215: BlockingRecv error: recv got EOF from 127.0.98.2:52215 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 24 07:10:14 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:14 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:14.725926 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:14.721280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.3:57565)
0924 07:10:14.721436 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:14.721449 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:14.721532 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:14.721640 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:14.721641 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:14.721796 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:14.721845 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:14.721847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:14.722459 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:14.722459 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:14.723000 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:14.723002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:14.723370 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:14.723371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:14.723412 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:14.723893 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:14.723898 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:14.725800 (+ 1902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":79,"threads_started":1}
W20260924 07:10:14.725939 9534 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:14.721367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57565 (local address 127.0.98.62:37663)
0924 07:10:14.721441 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:14.721443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:14.721519 (+ 76us) server_negotiation.cc:407] Connection header received
0924 07:10:14.721575 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:14.721576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:14.721588 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:14.721633 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:14.721904 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:14.722390 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:14.723086 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:14.723293 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:14.723356 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:14.725873 (+ 2517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57565: BlockingRecv error: recv got EOF from 127.0.98.3:57565 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 07:10:15 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:15 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:15.475864 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:15.471433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.1:39041)
0924 07:10:15.471575 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:15.471587 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:15.471651 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:15.471775 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:15.471776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:15.471932 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:15.471981 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:15.471982 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:15.472614 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:15.472615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:15.473199 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:15.473201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:15.473562 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:15.473563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:15.473602 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:15.474040 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:15.474045 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:15.475753 (+ 1708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":86,"threads_started":1}
W20260924 07:10:15.475885 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:15.471510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39041 (local address 127.0.98.62:37663)
0924 07:10:15.471602 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:15.471604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:15.471645 (+ 41us) server_negotiation.cc:407] Connection header received
0924 07:10:15.471693 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:15.471694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:15.471704 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:15.471761 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:15.472046 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:15.472551 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:15.473292 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:15.473498 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:15.473550 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:15.475825 (+ 2275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39041: BlockingRecv error: recv got EOF from 127.0.98.1:39041 (error 108)
Metrics: {"server-negotiator.queue_time_us":72,"thread_start_us":48,"threads_started":1}
Sep 24 07:10:15 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:15 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:15.601070 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:15.596418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.2:55923)
0924 07:10:15.596532 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:15.596544 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:15.596611 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:15.596775 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:15.596776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:15.596945 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:15.596994 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:15.596996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:15.597638 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:15.597638 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:15.598161 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:15.598162 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:15.598582 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:15.598583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:15.598638 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:15.599109 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:15.599114 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:15.601013 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":56,"threads_started":1}
W20260924 07:10:15.601181 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:15.596490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55923 (local address 127.0.98.62:37663)
0924 07:10:15.596574 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:15.596576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:15.596597 (+ 21us) server_negotiation.cc:407] Connection header received
0924 07:10:15.596662 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:15.596663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:15.596673 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:15.596715 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:15.597063 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:15.597568 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:15.598318 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:15.598504 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:15.598568 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:15.601106 (+ 2538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55923: BlockingRecv error: recv got EOF from 127.0.98.2:55923 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 07:10:15 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:15 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:15.731097 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:15.726461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.3:50225)
0924 07:10:15.726596 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:15.726608 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:15.726674 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:15.726845 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:15.726846 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:15.727025 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:15.727069 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:15.727071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:15.727757 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:15.727759 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:15.728305 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:15.728306 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:15.728667 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:15.728667 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:15.728724 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:15.729293 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:15.729299 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:15.731039 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":57,"threads_started":1}
W20260924 07:10:15.731236 9538 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:15.726616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50225 (local address 127.0.98.62:37663)
0924 07:10:15.726661 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:15.726664 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:15.726673 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:10:15.726745 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:15.726746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:15.726761 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:15.726822 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:15.727176 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:15.727672 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:15.728394 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:15.728581 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:15.728667 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:15.731151 (+ 2484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50225: BlockingRecv error: recv got EOF from 127.0.98.3:50225 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 24 07:10:16 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:16 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:16.481029 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:16.476500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.1:39905)
0924 07:10:16.476624 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:16.476636 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:16.476690 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:16.476869 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:16.476870 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:16.477050 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:16.477099 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:16.477101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:16.477862 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:16.477863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:16.478376 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:16.478377 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:16.478703 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:16.478704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:16.478743 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:16.479251 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:16.479256 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:16.480966 (+ 1710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":78,"threads_started":1}
W20260924 07:10:16.481099 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:16.476553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39905 (local address 127.0.98.62:37663)
0924 07:10:16.476667 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:16.476669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:16.476680 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:10:16.476776 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:16.476777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:16.476791 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:16.476860 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:16.477166 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:16.477795 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:16.478442 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:16.478637 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:16.478691 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:16.481048 (+ 2357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39905: BlockingRecv error: recv got EOF from 127.0.98.1:39905 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":72,"threads_started":1}
Sep 24 07:10:16 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:16 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:16.606206 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:16.601587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.2:42407)
0924 07:10:16.601744 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:16.601754 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:16.601815 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:16.601926 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:16.601927 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:16.602073 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:16.602118 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:16.602126 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:16.602846 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:16.602847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:16.603374 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:16.603375 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:16.603738 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:16.603739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:16.603783 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:16.604296 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:16.604302 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:16.606136 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":105,"threads_started":1}
W20260924 07:10:16.606271 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:16.601639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42407 (local address 127.0.98.62:37663)
0924 07:10:16.601710 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:16.601712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:16.601826 (+ 114us) server_negotiation.cc:407] Connection header received
0924 07:10:16.601856 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:16.601856 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:16.601867 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:16.601921 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:16.602230 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:16.602735 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:16.603446 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:16.603649 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:16.603706 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:16.606225 (+ 2519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42407: BlockingRecv error: recv got EOF from 127.0.98.2:42407 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 24 07:10:16 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:16 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:16.736366 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:16.731604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.3:45687)
0924 07:10:16.731802 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:16.731813 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:16.731904 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:16.732067 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:16.732067 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:16.732252 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:16.732298 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:16.732299 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:16.733045 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:16.733046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:16.733576 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:16.733584 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:16.733950 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:16.733951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:16.734005 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:16.734524 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:16.734529 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:16.736261 (+ 1732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":63,"threads_started":1}
W20260924 07:10:16.736404 9542 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:16.731680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45687 (local address 127.0.98.62:37663)
0924 07:10:16.731738 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:16.731740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:16.731887 (+ 147us) server_negotiation.cc:407] Connection header received
0924 07:10:16.731994 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:16.731995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:16.732009 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:16.732050 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:16.732390 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:16.732962 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:16.733674 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:16.733875 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:16.733953 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:16.736340 (+ 2387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45687: BlockingRecv error: recv got EOF from 127.0.98.3:45687 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 24 07:10:17 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:17 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:17.486335 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:17.481554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.1:39293)
0924 07:10:17.481710 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:17.481720 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:17.481781 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:17.481923 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:17.481924 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:17.482076 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:17.482125 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:17.482126 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:17.482862 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:17.482863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:17.483391 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:17.483393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:17.483729 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:17.483731 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:17.483829 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:17.484371 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:17.484376 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:17.486274 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":103,"threads_started":1}
W20260924 07:10:17.486442 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:17.481619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39293 (local address 127.0.98.62:37663)
0924 07:10:17.481733 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:17.481735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:17.481781 (+ 46us) server_negotiation.cc:407] Connection header received
0924 07:10:17.481836 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:17.481837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:17.481847 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:17.481910 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:17.482185 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:17.482792 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:17.483461 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:17.483644 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:17.483705 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:17.486359 (+ 2654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39293: BlockingRecv error: recv got EOF from 127.0.98.1:39293 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":49,"threads_started":1}
Sep 24 07:10:17 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:17 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:17.619911 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:17.606758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.2:57411)
0924 07:10:17.606906 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:17.606919 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:17.606984 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:17.607115 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:17.607116 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:17.607271 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:17.607350 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:17.607352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:17.608070 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:17.608071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:17.608605 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:17.608606 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:17.609014 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:17.609015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:17.609064 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:17.618022 (+ 8958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:17.618028 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:17.619824 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":93,"threads_started":1}
W20260924 07:10:17.620087 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:17.606890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57411 (local address 127.0.98.62:37663)
0924 07:10:17.606934 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:17.606936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:17.606968 (+ 32us) server_negotiation.cc:407] Connection header received
0924 07:10:17.607038 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:17.607039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:17.607050 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:17.607098 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:17.607469 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:17.607989 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:17.608683 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:17.608927 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:17.608986 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:17.620016 (+ 11030us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57411: BlockingRecv error: recv got EOF from 127.0.98.2:57411 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 24 07:10:17 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:17 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:17.741470 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:17.736894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.3:49899)
0924 07:10:17.737029 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:17.737039 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:17.737091 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:17.737224 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:17.737225 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:17.737385 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:17.737433 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:17.737435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:17.738189 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:17.738191 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:17.738723 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:17.738725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:17.739051 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:17.739052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:17.739093 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:17.739554 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:17.739558 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:17.741370 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":77,"threads_started":1}
W20260924 07:10:17.741521 9546 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:17.736993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49899 (local address 127.0.98.62:37663)
0924 07:10:17.737052 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:17.737054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:17.737081 (+ 27us) server_negotiation.cc:407] Connection header received
0924 07:10:17.737143 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:17.737143 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:17.737154 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:17.737205 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:17.737556 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:17.738088 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:17.738806 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:17.738983 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:17.739037 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:17.741449 (+ 2412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49899: BlockingRecv error: recv got EOF from 127.0.98.3:49899 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 24 07:10:18 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:18 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:18.491509 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:18.486783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.1:32967)
0924 07:10:18.486907 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:18.486919 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:18.486984 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:18.487115 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:18.487115 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:18.487270 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:18.487333 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:18.487334 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:18.488010 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:18.488010 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:18.488546 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:18.488547 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:18.489044 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:18.489046 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:18.489115 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:18.489656 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:18.489660 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:18.491400 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":57,"threads_started":1}
W20260924 07:10:18.491549 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:18.486878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32967 (local address 127.0.98.62:37663)
0924 07:10:18.486997 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:18.486999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:18.487009 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:10:18.487031 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:18.487032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:18.487041 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:18.487099 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:18.487408 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:18.487942 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:18.488629 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:18.488958 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:18.489042 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:18.491475 (+ 2433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:32967: BlockingRecv error: recv got EOF from 127.0.98.1:32967 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
Sep 24 07:10:18 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:18 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:18.625262 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:18.620435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.2:56625)
0924 07:10:18.620560 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:18.620578 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:18.620646 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:18.620827 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:18.620828 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:18.621009 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:18.621054 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:18.621056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:18.621757 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:18.621758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:18.622285 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:18.622286 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:18.622637 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:18.622638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:18.622706 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:18.623199 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:18.623204 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:18.625151 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":75,"threads_started":1}
W20260924 07:10:18.625288 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:18.620600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56625 (local address 127.0.98.62:37663)
0924 07:10:18.620671 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:18.620673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:18.620691 (+ 18us) server_negotiation.cc:407] Connection header received
0924 07:10:18.620716 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:18.620717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:18.620753 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:18.620820 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:18.621142 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:18.621638 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:18.622369 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:18.622563 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:18.622639 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:18.625226 (+ 2587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56625: BlockingRecv error: recv got EOF from 127.0.98.2:56625 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 24 07:10:18 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:18 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:18.754563 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:18.742119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.3:33049)
0924 07:10:18.742293 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:18.742305 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:18.742396 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:18.742543 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:18.742543 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:18.742723 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:18.742768 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:18.742770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:18.743468 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:18.743469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:18.744017 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:18.744019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:18.744376 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:18.744377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:18.744442 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:18.752835 (+ 8393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:18.752840 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:18.754497 (+ 1657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":96,"threads_started":1}
W20260924 07:10:18.754689 9550 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:18.742222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33049 (local address 127.0.98.62:37663)
0924 07:10:18.742291 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:18.742293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:18.742378 (+ 85us) server_negotiation.cc:407] Connection header received
0924 07:10:18.742451 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:18.742451 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:18.742466 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:18.742536 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:18.742886 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:18.743386 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:18.744105 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:18.744293 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:18.744372 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:18.754589 (+ 10217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33049: BlockingRecv error: recv got EOF from 127.0.98.3:33049 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 07:10:19 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:19 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:19.497293 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:19.491941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.1:45717)
0924 07:10:19.492083 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:19.492092 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:19.492141 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:19.492252 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:19.492252 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:19.492405 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:19.492454 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:19.492455 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:19.493174 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:19.493175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:19.493691 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:19.493692 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:19.494050 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:19.494050 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:19.494088 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:19.495423 (+ 1335us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:19.495428 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:19.497197 (+ 1769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
W20260924 07:10:19.497344 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:19.492003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45717 (local address 127.0.98.62:37663)
0924 07:10:19.492142 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:19.492144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:19.492170 (+ 26us) server_negotiation.cc:407] Connection header received
0924 07:10:19.492189 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:19.492190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:19.492200 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:19.492247 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:19.492515 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:19.493103 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:19.493785 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:19.493979 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:19.494040 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:19.497272 (+ 3232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45717: BlockingRecv error: recv got EOF from 127.0.98.1:45717 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":72,"threads_started":1}
Sep 24 07:10:19 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:19 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:19.631352 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:19.625771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.2:50533)
0924 07:10:19.625927 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:19.625946 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:19.626005 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:19.626139 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:19.626139 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:19.626359 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:19.626407 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:19.626409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:19.627142 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:19.627143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:19.627679 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:19.627681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:19.628032 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:19.628033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:19.628073 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:19.629543 (+ 1470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:19.629548 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:19.631251 (+ 1703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":83,"threads_started":1}
W20260924 07:10:19.631430 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:19.625834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50533 (local address 127.0.98.62:37663)
0924 07:10:19.625948 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:19.625951 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:19.625993 (+ 42us) server_negotiation.cc:407] Connection header received
0924 07:10:19.626066 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:19.626067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:19.626078 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:19.626122 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:19.626485 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:19.627070 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:19.627769 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:19.627958 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:19.628025 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:19.631343 (+ 3318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50533: BlockingRecv error: recv got EOF from 127.0.98.2:50533 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 24 07:10:19 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:19 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:19.760551 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:19.755061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.3:33631)
0924 07:10:19.755199 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:19.755210 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:19.755271 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:19.755391 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:19.755392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:19.755549 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:19.755599 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:19.755601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:19.756294 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:19.756295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:19.756959 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:19.756961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:19.757404 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:19.757405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:19.757461 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:19.758548 (+ 1087us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:19.758553 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:19.760457 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":67,"threads_started":1}
W20260924 07:10:19.760594 9554 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:19.755109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33631 (local address 127.0.98.62:37663)
0924 07:10:19.755160 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:19.755162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:19.755259 (+ 97us) server_negotiation.cc:407] Connection header received
0924 07:10:19.755324 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:19.755325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:19.755337 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:19.755376 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:19.755662 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:19.756228 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:19.757054 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:19.757318 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:19.757409 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:19.760533 (+ 3124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33631: BlockingRecv error: recv got EOF from 127.0.98.3:33631 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 24 07:10:20 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:20 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:20.503412 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:20.497803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.1:45553)
0924 07:10:20.497991 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:20.498003 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:20.498050 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:20.498186 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:20.498187 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:20.498359 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:20.498422 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:20.498424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:20.499082 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:20.499083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:20.499623 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:20.499624 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:20.499952 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:20.499952 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:20.499990 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:20.501396 (+ 1406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:20.501402 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:20.503312 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":100,"threads_started":1}
W20260924 07:10:20.503469 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:20.497917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45553 (local address 127.0.98.62:37663)
0924 07:10:20.498026 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:20.498028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:20.498051 (+ 23us) server_negotiation.cc:407] Connection header received
0924 07:10:20.498104 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:20.498105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:20.498115 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:20.498165 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:20.498490 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:20.499009 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:20.499695 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:20.499887 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:20.499940 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:20.503396 (+ 3456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45553: BlockingRecv error: recv got EOF from 127.0.98.1:45553 (error 108)
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":55,"threads_started":1}
Sep 24 07:10:20 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:20 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:20.637399 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:20.631910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.2:49397)
0924 07:10:20.632061 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:20.632079 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:20.632159 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:20.632323 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:20.632324 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:20.632505 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:20.632553 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:20.632555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:20.633320 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:20.633321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:20.633848 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:20.633849 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:20.634208 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:20.634208 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:20.634267 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:20.635531 (+ 1264us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:20.635536 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:20.637307 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":75,"threads_started":1}
W20260924 07:10:20.637436 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:20.631944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49397 (local address 127.0.98.62:37663)
0924 07:10:20.632028 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:20.632030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:20.632140 (+ 110us) server_negotiation.cc:407] Connection header received
0924 07:10:20.632216 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:20.632217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:20.632231 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:20.632305 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:20.632638 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:20.633229 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:20.633942 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:20.634136 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:20.634211 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:20.637378 (+ 3167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49397: BlockingRecv error: recv got EOF from 127.0.98.2:49397 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 07:10:20 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:20 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:20.766490 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:20.761047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.3:54973)
0924 07:10:20.761210 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:20.761230 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:20.761286 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:20.761474 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:20.761474 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:20.761687 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:20.761754 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:20.761757 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:20.762519 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:20.762521 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:20.763049 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:20.763050 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:20.763369 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:20.763370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:20.763412 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:20.764685 (+ 1273us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:20.764690 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:20.766430 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":103,"threads_started":1}
W20260924 07:10:20.766667 9558 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:20.761170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54973 (local address 127.0.98.62:37663)
0924 07:10:20.761226 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:20.761228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:20.761273 (+ 45us) server_negotiation.cc:407] Connection header received
0924 07:10:20.761344 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:20.761345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:20.761385 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:20.761461 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:20.761841 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:20.762430 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:20.763124 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:20.763299 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:20.763348 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:20.766568 (+ 3220us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54973: BlockingRecv error: recv got EOF from 127.0.98.3:54973 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 24 07:10:21 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:21 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:21.509284 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:21.503894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.1:36807)
0924 07:10:21.504051 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:21.504065 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:21.504129 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:21.504284 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:21.504285 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:21.504441 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:21.504491 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:21.504493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:21.505226 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:21.505227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:21.505752 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:21.505753 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:21.506134 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:21.506134 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:21.506180 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:21.507475 (+ 1295us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:21.507480 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:21.509227 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":74,"threads_started":1}
W20260924 07:10:21.509464 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:21.504038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36807 (local address 127.0.98.62:37663)
0924 07:10:21.504171 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:21.504173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:21.504180 (+ 7us) server_negotiation.cc:407] Connection header received
0924 07:10:21.504200 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:21.504201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:21.504211 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:21.504267 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:21.504556 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:21.505153 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:21.505866 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:21.506060 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:21.506114 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:21.509365 (+ 3251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36807: BlockingRecv error: recv got EOF from 127.0.98.1:36807 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
Sep 24 07:10:21 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:21 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:21.643051 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:21.637888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.2:58067)
0924 07:10:21.638038 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:21.638048 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:21.638106 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:21.638225 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:21.638226 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:21.638381 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:21.638429 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:21.638431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:21.639207 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:21.639208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:21.639966 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:21.639969 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:21.640280 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:21.640281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:21.640318 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:21.641327 (+ 1009us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:21.641331 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:21.642981 (+ 1650us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":92,"threads_started":1}
W20260924 07:10:21.643157 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:21.637936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58067 (local address 127.0.98.62:37663)
0924 07:10:21.638004 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:21.638006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:21.638113 (+ 107us) server_negotiation.cc:407] Connection header received
0924 07:10:21.638155 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:21.638156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:21.638168 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:21.638210 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:21.638492 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:21.639135 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:21.640033 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:21.640218 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:21.640271 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:21.643064 (+ 2793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58067: BlockingRecv error: recv got EOF from 127.0.98.2:58067 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 07:10:21 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:21 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:21.772622 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:21.767061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.3:58015)
0924 07:10:21.767205 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:21.767223 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:21.767282 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:21.767409 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:21.767410 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:21.767566 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:21.767614 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:21.767616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:21.768242 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:21.768243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:21.769086 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:21.769089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:21.769476 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:21.769478 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:21.769523 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:21.770779 (+ 1256us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:21.770784 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:21.772526 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":98,"threads_started":1}
W20260924 07:10:21.772668 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:21.767115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58015 (local address 127.0.98.62:37663)
0924 07:10:21.767184 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:21.767191 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:21.767280 (+ 89us) server_negotiation.cc:407] Connection header received
0924 07:10:21.767325 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:21.767325 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:21.767339 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:21.767391 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:21.767672 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:21.768171 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:21.769179 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:21.769372 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:21.769448 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:21.772603 (+ 3155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58015: BlockingRecv error: recv got EOF from 127.0.98.3:58015 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 24 07:10:22 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:22 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:22.515529 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:22.509952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.1:43059)
0924 07:10:22.510103 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:22.510113 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:22.510169 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:22.510310 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:22.510310 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:22.510476 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:22.510523 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:22.510525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:22.511276 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:22.511278 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:22.511861 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:22.511863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:22.512240 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:22.512241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:22.512283 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:22.513692 (+ 1409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:22.513696 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:22.515473 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
W20260924 07:10:22.515697 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:22.510022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43059 (local address 127.0.98.62:37663)
0924 07:10:22.510166 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:22.510168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:22.510177 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:10:22.510218 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:22.510218 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:22.510229 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:22.510291 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:22.510628 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:22.511182 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:22.511973 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:22.512162 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:22.512220 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:22.515584 (+ 3364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43059: BlockingRecv error: recv got EOF from 127.0.98.1:43059 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":87,"threads_started":1}
Sep 24 07:10:22 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:22 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:22.658300 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:22.643496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.2:47169)
0924 07:10:22.643680 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:22.643692 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:22.643780 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:22.643927 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:22.643927 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:22.644132 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:22.644178 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:22.644180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:22.644917 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:22.644918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:22.645454 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:22.645456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:22.645833 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:22.645834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:22.645899 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:22.656335 (+ 10436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:22.656341 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:22.658189 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":90,"threads_started":1}
W20260924 07:10:22.658347 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:22.643582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47169 (local address 127.0.98.62:37663)
0924 07:10:22.643663 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:22.643666 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:22.643761 (+ 95us) server_negotiation.cc:407] Connection header received
0924 07:10:22.643855 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:22.643856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:22.643869 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:22.643923 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:22.644264 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:22.644838 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:22.645543 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:22.645747 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:22.645832 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:22.658271 (+ 12439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47169: BlockingRecv error: recv got EOF from 127.0.98.2:47169 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 24 07:10:22 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:22 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:22.777894 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:22.773088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.3:55787)
0924 07:10:22.773261 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:22.773273 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:22.773362 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:22.773512 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:22.773513 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:22.773693 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:22.773739 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:22.773741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:22.774441 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:22.774442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:22.774961 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:22.774962 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:22.775334 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:22.775335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:22.775398 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:22.775916 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:22.775921 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:22.777795 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":104,"threads_started":1}
W20260924 07:10:22.777941 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:22.773159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55787 (local address 127.0.98.62:37663)
0924 07:10:22.773253 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:22.773256 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:22.773345 (+ 89us) server_negotiation.cc:407] Connection header received
0924 07:10:22.773447 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:22.773447 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:22.773461 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:22.773508 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:22.773824 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:22.774341 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:22.775051 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:22.775251 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:22.775324 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:22.777871 (+ 2547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55787: BlockingRecv error: recv got EOF from 127.0.98.3:55787 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 24 07:10:23 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:23 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:23.520530 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.515994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.1:43629)
0924 07:10:23.516148 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:23.516158 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:23.516209 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:23.516345 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:23.516346 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:23.516517 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:23.516566 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:23.516568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.517199 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.517200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:23.517726 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:23.517727 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.518080 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.518085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:23.518158 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:23.518729 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:23.518734 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:23.520435 (+ 1701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":88,"threads_started":1}
W20260924 07:10:23.520591 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:23.516077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43629 (local address 127.0.98.62:37663)
0924 07:10:23.516192 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:23.516194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:23.516210 (+ 16us) server_negotiation.cc:407] Connection header received
0924 07:10:23.516253 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:23.516254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:23.516270 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:23.516326 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:23.516631 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.517139 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.517794 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.517983 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.518034 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:23.520517 (+ 2483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43629: BlockingRecv error: recv got EOF from 127.0.98.1:43629 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":49,"threads_started":1}
Sep 24 07:10:23 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:23 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:23.663444 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.658870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.2:59759)
0924 07:10:23.659003 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:23.659015 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:23.659078 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:23.659203 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:23.659204 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:23.659368 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:23.659417 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:23.659419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.660034 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.660035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:23.660562 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:23.660563 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.661145 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.661146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:23.661214 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:23.661650 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:23.661654 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:23.663387 (+ 1733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":75,"threads_started":1}
W20260924 07:10:23.663574 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:23.658937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59759 (local address 127.0.98.62:37663)
0924 07:10:23.658998 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:23.659000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:23.659079 (+ 79us) server_negotiation.cc:407] Connection header received
0924 07:10:23.659120 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:23.659121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:23.659132 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:23.659185 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:23.659477 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.659957 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.660663 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.661023 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.661110 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:23.663492 (+ 2382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59759: BlockingRecv error: recv got EOF from 127.0.98.2:59759 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 24 07:10:23 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:23 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:23.783104 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:23.778475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.3:44725)
0924 07:10:23.778597 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:23.778611 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:23.778668 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:23.778818 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:23.778819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:23.779061 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:23.779108 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:23.779110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.779728 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.779729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:23.780276 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:23.780278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.780671 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.780671 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:23.780720 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:23.781293 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:23.781297 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:23.783032 (+ 1735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":66,"threads_started":1}
W20260924 07:10:23.783244 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:23.778584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44725 (local address 127.0.98.62:37663)
0924 07:10:23.778640 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:23.778642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:23.778654 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:10:23.778739 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:23.778740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:23.778754 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:23.778799 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:23.779174 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.779656 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.780391 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:23.780578 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:23.780643 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:23.783106 (+ 2463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44725: BlockingRecv error: recv got EOF from 127.0.98.3:44725 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 24 07:10:24 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:24 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:24.525822 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:24.521013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.1:37825)
0924 07:10:24.521145 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:24.521156 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:24.521206 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:24.521369 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:24.521370 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:24.521529 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:24.521578 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:24.521579 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:24.522214 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:24.522215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:24.522950 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:24.522959 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:24.523317 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:24.523318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:24.523357 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:24.523843 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:24.523849 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:24.525760 (+ 1911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":77,"threads_started":1}
W20260924 07:10:24.525951 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:24.521098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37825 (local address 127.0.98.62:37663)
0924 07:10:24.521230 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:24.521232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:24.521241 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:10:24.521279 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:24.521279 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:24.521289 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:24.521353 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:24.521645 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:24.522140 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:24.523053 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:24.523242 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:24.523304 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:24.525845 (+ 2541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37825: BlockingRecv error: recv got EOF from 127.0.98.1:37825 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
Sep 24 07:10:24 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:24 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:24.668645 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:24.663872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.2:54343)
0924 07:10:24.664048 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:24.664065 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:24.664153 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:24.664319 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:24.664320 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:24.664500 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:24.664546 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:24.664548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:24.665283 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:24.665284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:24.665837 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:24.665839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:24.666226 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:24.666227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:24.666284 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:24.666792 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:24.666797 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:24.668553 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":104,"threads_started":1}
W20260924 07:10:24.668690 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:24.663939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54343 (local address 127.0.98.62:37663)
0924 07:10:24.664015 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:24.664018 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:24.664133 (+ 115us) server_negotiation.cc:407] Connection header received
0924 07:10:24.664209 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:24.664210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:24.664224 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:24.664307 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:24.664628 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:24.665200 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:24.665941 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:24.666142 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:24.666228 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:24.668626 (+ 2398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54343: BlockingRecv error: recv got EOF from 127.0.98.2:54343 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 24 07:10:24 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:24 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:24.787889 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:24.783503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.3:37813)
0924 07:10:24.783650 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:24.783666 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:24.783741 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:24.783878 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:24.783879 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:24.784039 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:24.784087 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:24.784089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:24.784718 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:24.784719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:24.785296 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:24.785298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:24.785634 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:24.785635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:24.785673 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:24.786125 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:24.786130 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:24.787827 (+ 1697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":102,"threads_started":1}
W20260924 07:10:24.788065 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:24.783597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37813 (local address 127.0.98.62:37663)
0924 07:10:24.783654 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:24.783656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:24.783722 (+ 66us) server_negotiation.cc:407] Connection header received
0924 07:10:24.783789 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:24.783790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:24.783802 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:24.783864 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:24.784148 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:24.784650 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:24.785380 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:24.785569 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:24.785625 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:24.787995 (+ 2370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37813: BlockingRecv error: recv got EOF from 127.0.98.3:37813 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 24 07:10:25 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:25 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:25.531746 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:25.526317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.1:54603)
0924 07:10:25.526455 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:25.526467 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:25.526517 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:25.526648 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:25.526649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:25.526811 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:25.526886 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:25.526888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:25.527799 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:25.527800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:25.528337 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:25.528339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:25.528971 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:25.528972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:25.529023 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:25.529493 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:25.529497 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:25.531682 (+ 2185us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
W20260924 07:10:25.531890 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:25.526369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54603 (local address 127.0.98.62:37663)
0924 07:10:25.526493 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:25.526495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:25.526548 (+ 53us) server_negotiation.cc:407] Connection header received
0924 07:10:25.526568 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:25.526569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:25.526579 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:25.526630 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:25.527021 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:25.527723 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:25.528438 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:25.528722 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:25.528812 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:25.531773 (+ 2961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54603: BlockingRecv error: recv got EOF from 127.0.98.1:54603 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":72,"threads_started":1}
Sep 24 07:10:25 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:25 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:25.673596 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:25.669138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.2:39277)
0924 07:10:25.669300 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:25.669311 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:25.669390 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:25.669510 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:25.669510 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:25.669669 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:25.669715 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:25.669717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:25.670348 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:25.670349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:25.670865 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:25.670866 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:25.671206 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:25.671207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:25.671269 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:25.671717 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:25.671722 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:25.673535 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":115,"threads_started":1}
W20260924 07:10:25.673717 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:25.669204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39277 (local address 127.0.98.62:37663)
0924 07:10:25.669283 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:25.669287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:25.669386 (+ 99us) server_negotiation.cc:407] Connection header received
0924 07:10:25.669446 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:25.669447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:25.669458 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:25.669495 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:25.669770 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:25.670268 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:25.670940 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:25.671121 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:25.671177 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:25.673620 (+ 2443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39277: BlockingRecv error: recv got EOF from 127.0.98.2:39277 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 07:10:25 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:25 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:25.793094 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:25.788411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.3:39067)
0924 07:10:25.788568 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:25.788585 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:25.788667 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:25.788822 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:25.788823 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:25.789009 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:25.789053 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:25.789055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:25.789725 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:25.789727 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:25.790257 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:25.790259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:25.790609 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:25.790610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:25.790671 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:25.791158 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:25.791162 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:25.793002 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":87,"threads_started":1}
W20260924 07:10:25.793131 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:25.788466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39067 (local address 127.0.98.62:37663)
0924 07:10:25.788535 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:25.788538 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:25.788649 (+ 111us) server_negotiation.cc:407] Connection header received
0924 07:10:25.788721 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:25.788722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:25.788760 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:25.788807 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:25.789134 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:25.789642 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:25.790339 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:25.790535 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:25.790612 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:25.793070 (+ 2458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39067: BlockingRecv error: recv got EOF from 127.0.98.3:39067 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 24 07:10:26 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:26 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:26.536723 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.532216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.1:44999)
0924 07:10:26.532360 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:26.532370 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:26.532424 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:26.532547 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:26.532548 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:26.532704 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:26.532805 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:26.532807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.533441 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.533443 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:26.533980 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:26.533982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.534364 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.534365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:26.534409 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:26.534953 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:26.534958 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:26.536658 (+ 1700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
W20260924 07:10:26.536931 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:26.532291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44999 (local address 127.0.98.62:37663)
0924 07:10:26.532421 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:26.532423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:26.532433 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:10:26.532470 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:26.532470 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:26.532481 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:26.532541 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:26.532863 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.533363 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.534095 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.534290 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.534356 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:26.536822 (+ 2466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44999: BlockingRecv error: recv got EOF from 127.0.98.1:44999 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":78,"threads_started":1}
Sep 24 07:10:26 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:26 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:26.679031 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.674075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.2:40989)
0924 07:10:26.674203 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:26.674218 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:26.674266 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:26.674420 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:26.674420 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:26.674583 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:26.674630 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:26.674632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.675505 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.675506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:26.676185 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:26.676189 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.676634 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.676635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:26.676676 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:26.677165 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:26.677169 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:26.678972 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
W20260924 07:10:26.679132 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:26.674182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40989 (local address 127.0.98.62:37663)
0924 07:10:26.674252 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:26.674254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:26.674264 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:10:26.674329 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:26.674330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:26.674341 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:26.674393 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:26.674698 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.675413 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.676272 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.676566 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.676618 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:26.679066 (+ 2448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40989: BlockingRecv error: recv got EOF from 127.0.98.2:40989 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 24 07:10:26 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:26 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:26.798477 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:26.793676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.3:33055)
0924 07:10:26.793816 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:26.793826 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:26.793889 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:26.794102 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:26.794103 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:26.794318 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:26.794373 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:26.794375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.795031 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.795032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:26.795559 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:26.795561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.795887 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.795888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:26.795930 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:26.796373 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:26.796377 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:26.798375 (+ 1998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":86,"threads_started":1}
W20260924 07:10:26.798538 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:26.793732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33055 (local address 127.0.98.62:37663)
0924 07:10:26.793782 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:26.793784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:26.793899 (+ 115us) server_negotiation.cc:407] Connection header received
0924 07:10:26.793990 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:26.793991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:26.794008 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:26.794073 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:26.794459 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.794952 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.795637 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:26.795818 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:26.795879 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:26.798478 (+ 2599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33055: BlockingRecv error: recv got EOF from 127.0.98.3:33055 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 24 07:10:27 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:27 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:27.542155 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:27.537642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.1:53041)
0924 07:10:27.537808 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:27.537818 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:27.537876 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:27.537993 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:27.537994 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:27.538155 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:27.538205 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:27.538206 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:27.538879 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:27.538880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:27.539415 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:27.539416 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:27.539742 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:27.539742 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:27.539781 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:27.540215 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:27.540220 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:27.542081 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":97,"threads_started":1}
W20260924 07:10:27.542339 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:27.537727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53041 (local address 127.0.98.62:37663)
0924 07:10:27.537833 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:27.537835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:27.537885 (+ 50us) server_negotiation.cc:407] Connection header received
0924 07:10:27.537920 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:27.537921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:27.537930 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:27.537988 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:27.538274 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:27.538805 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:27.539491 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:27.539677 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:27.539729 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:27.542235 (+ 2506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53041: BlockingRecv error: recv got EOF from 127.0.98.1:53041 (error 108)
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":47,"threads_started":1}
Sep 24 07:10:27 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:27 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:27.694451 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:27.679526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.2:33401)
0924 07:10:27.679677 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:27.679687 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:27.679748 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:27.679879 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:27.679880 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:27.680043 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:27.680090 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:27.680092 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:27.680772 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:27.680773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:27.681343 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:27.681344 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:27.681677 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:27.681678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:27.681717 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:27.692556 (+ 10839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:27.692561 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:27.694375 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":83,"threads_started":1}
W20260924 07:10:27.694641 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:27.679593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33401 (local address 127.0.98.62:37663)
0924 07:10:27.679670 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:27.679672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:27.679749 (+ 77us) server_negotiation.cc:407] Connection header received
0924 07:10:27.679803 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:27.679804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:27.679815 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:27.679855 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:27.680173 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:27.680661 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:27.681421 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:27.681607 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:27.681663 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:27.694573 (+ 12910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33401: BlockingRecv error: recv got EOF from 127.0.98.2:33401 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 07:10:27 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:27 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:27.803581 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:27.799024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.3:36645)
0924 07:10:27.799161 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:27.799172 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:27.799238 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:27.799377 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:27.799377 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:27.799544 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:27.799594 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:27.799595 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:27.800240 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:27.800241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:27.800808 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:27.800809 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:27.801179 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:27.801180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:27.801222 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:27.801733 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:27.801738 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:27.803519 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
W20260924 07:10:27.803653 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:27.799072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36645 (local address 127.0.98.62:37663)
0924 07:10:27.799142 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:27.799144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:27.799240 (+ 96us) server_negotiation.cc:407] Connection header received
0924 07:10:27.799297 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:27.799297 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:27.799311 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:27.799353 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:27.799654 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:27.800160 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:27.800925 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:27.801101 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:27.801155 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:27.803607 (+ 2452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36645: BlockingRecv error: recv got EOF from 127.0.98.3:36645 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 24 07:10:28 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:28 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:28.547268 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:28.542710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.1:51991)
0924 07:10:28.542843 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:28.542854 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:28.542908 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:28.543099 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:28.543100 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:28.543269 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:28.543316 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:28.543318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:28.543974 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:28.543975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:28.544494 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:28.544496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:28.544928 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:28.544929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:28.544978 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:28.545401 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:28.545406 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:28.547212 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
W20260924 07:10:28.547394 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:28.542804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51991 (local address 127.0.98.62:37663)
0924 07:10:28.542923 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:28.542925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:28.542934 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:10:28.542957 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:28.542958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:28.542976 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:28.543057 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:28.543396 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:28.543884 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:28.544584 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:28.544805 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:28.544895 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:28.547309 (+ 2414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51991: BlockingRecv error: recv got EOF from 127.0.98.1:51991 (error 108)
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":66,"threads_started":1}
Sep 24 07:10:28 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 07:10:28 dist-test-slave-14ww krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 07:10:28.699429 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:28.694978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.2:45361)
0924 07:10:28.695111 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:28.695122 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:28.695186 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:28.695296 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:28.695296 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:28.695447 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:28.695494 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:28.695496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:28.696229 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:28.696230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:28.696806 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:28.696808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:28.697178 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:28.697178 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:28.697217 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:28.697641 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:28.697645 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:28.699373 (+ 1728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
W20260924 07:10:28.699559 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:28.695047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45361 (local address 127.0.98.62:37663)
0924 07:10:28.695114 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:28.695116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:28.695190 (+ 74us) server_negotiation.cc:407] Connection header received
0924 07:10:28.695226 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:28.695226 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:28.695236 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:28.695286 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:28.695552 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:28.696133 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:28.696884 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:28.697104 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:28.697168 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:28.699475 (+ 2307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45361: BlockingRecv error: recv got EOF from 127.0.98.2:45361 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 07:10:28.715657 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 9115
I20260924 07:10:28.720300 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 9267
I20260924 07:10:28.724668 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 9399
I20260924 07:10:28.728827 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 9045
2026-09-24T07:10:28Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17491 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T07:10:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:10:28Z Disabled control of system clock
I20260924 07:10:28.755820 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35561
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35561
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 07:10:28.834960 9595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:28.835145 9595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:28.835165 9595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:28.836249 9595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:10:28.836284 9595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:28.836297 9595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:28.836309 9595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:10:28.836321 9595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:10:28.837576 9595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35561
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:10:28.837752 9595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:28.837980 9595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:28.843022 9601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:28.843092 9603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:28.843030 9600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:28.843060 9595 server_base.cc:1034] running on GCE node
I20260924 07:10:28.843405 9595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:28.843670 9595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:28.844797 9595 hybrid_clock.cc:648] HybridClock initialized: now 1790233828844787 us; error 44 us; skew 500 ppm
I20260924 07:10:28.845923 9595 webserver.cc:460] Webserver started at http://127.0.98.62:41875/ using document root <none> and password file <none>
I20260924 07:10:28.846160 9595 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:28.846225 9595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:28.846369 9595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:28.847357 9595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "83707d3956b54931bd9a1731cb9ac5e0"
format_stamp: "Formatted at 2026-09-24 07:10:28 on dist-test-slave-14ww"
server_key: "0ebd43e2cd294e170d4c4b1c462dfebd"
server_key_iv: "192ffbc68eea9a97dd9f46c0e16c4ecd"
server_key_version: "encryptionkey@0"
I20260924 07:10:28.847690 9595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "83707d3956b54931bd9a1731cb9ac5e0"
format_stamp: "Formatted at 2026-09-24 07:10:28 on dist-test-slave-14ww"
server_key: "0ebd43e2cd294e170d4c4b1c462dfebd"
server_key_iv: "192ffbc68eea9a97dd9f46c0e16c4ecd"
server_key_version: "encryptionkey@0"
I20260924 07:10:28.848980 9595 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 07:10:28.849639 9609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:28.849799 9595 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:10:28.849896 9595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "83707d3956b54931bd9a1731cb9ac5e0"
format_stamp: "Formatted at 2026-09-24 07:10:28 on dist-test-slave-14ww"
server_key: "0ebd43e2cd294e170d4c4b1c462dfebd"
server_key_iv: "192ffbc68eea9a97dd9f46c0e16c4ecd"
server_key_version: "encryptionkey@0"
I20260924 07:10:28.849970 9595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:28.869865 9595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:28.870136 9595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:28.870280 9595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:28.874087 9595 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35561
I20260924 07:10:28.874115 9661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35561 every 8 connection(s)
I20260924 07:10:28.874431 9595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:10:28.875052 9662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:28.877661 9662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83707d3956b54931bd9a1731cb9ac5e0: Bootstrap starting.
I20260924 07:10:28.878186 9662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83707d3956b54931bd9a1731cb9ac5e0: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:28.878410 9662 log.cc:824] T 00000000000000000000000000000000 P 83707d3956b54931bd9a1731cb9ac5e0: Log is configured to *not* fsync() on all Append() calls
I20260924 07:10:28.879055 9662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83707d3956b54931bd9a1731cb9ac5e0: No bootstrap required, opened a new log
I20260924 07:10:28.879771 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 9595
I20260924 07:10:28.879871 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:10:28.880057 392 external_mini_cluster.cc:1428] Setting key 249769c8e703643d276661366c07d497
I20260924 07:10:28.881158 9662 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83707d3956b54931bd9a1731cb9ac5e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83707d3956b54931bd9a1731cb9ac5e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35561 } }
I20260924 07:10:28.881316 9662 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83707d3956b54931bd9a1731cb9ac5e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:28.881352 9662 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83707d3956b54931bd9a1731cb9ac5e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83707d3956b54931bd9a1731cb9ac5e0, State: Initialized, Role: FOLLOWER
I20260924 07:10:28.881484 9662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83707d3956b54931bd9a1731cb9ac5e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83707d3956b54931bd9a1731cb9ac5e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35561 } }
I20260924 07:10:28.881573 9662 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83707d3956b54931bd9a1731cb9ac5e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:10:28.881624 9662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83707d3956b54931bd9a1731cb9ac5e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:10:28.881675 9662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83707d3956b54931bd9a1731cb9ac5e0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:28.882184 9662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83707d3956b54931bd9a1731cb9ac5e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83707d3956b54931bd9a1731cb9ac5e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35561 } }
I20260924 07:10:28.882266 9662 leader_election.cc:302] T 00000000000000000000000000000000 P 83707d3956b54931bd9a1731cb9ac5e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83707d3956b54931bd9a1731cb9ac5e0; no voters:
I20260924 07:10:28.882431 9662 leader_election.cc:288] T 00000000000000000000000000000000 P 83707d3956b54931bd9a1731cb9ac5e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:10:28.882455 9667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83707d3956b54931bd9a1731cb9ac5e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:10:28.882556 9667 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83707d3956b54931bd9a1731cb9ac5e0 [term 1 LEADER]: Becoming Leader. State: Replica: 83707d3956b54931bd9a1731cb9ac5e0, State: Running, Role: LEADER
I20260924 07:10:28.882633 9667 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83707d3956b54931bd9a1731cb9ac5e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83707d3956b54931bd9a1731cb9ac5e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35561 } }
I20260924 07:10:28.882681 9662 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83707d3956b54931bd9a1731cb9ac5e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:10:28.882952 9667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83707d3956b54931bd9a1731cb9ac5e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83707d3956b54931bd9a1731cb9ac5e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83707d3956b54931bd9a1731cb9ac5e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35561 } } }
I20260924 07:10:28.883025 9667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83707d3956b54931bd9a1731cb9ac5e0 [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:28.883141 9667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83707d3956b54931bd9a1731cb9ac5e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83707d3956b54931bd9a1731cb9ac5e0. Latest consensus state: current_term: 1 leader_uuid: "83707d3956b54931bd9a1731cb9ac5e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83707d3956b54931bd9a1731cb9ac5e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35561 } } }
I20260924 07:10:28.883209 9667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83707d3956b54931bd9a1731cb9ac5e0 [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:28.883502 9672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:10:28.884034 9672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:10:28.885743 9672 catalog_manager.cc:1347] Generated new cluster ID: b2a2ca09717c4b51ac53eb23973b5ed2
I20260924 07:10:28.885807 9672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:10:28.886655 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:28.880591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60222 (local address 127.0.98.62:35561)
0924 07:10:28.880834 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:28.880841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:28.880868 (+ 27us) server_negotiation.cc:407] Connection header received
0924 07:10:28.881139 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:28.881145 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:28.881199 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:28.881307 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:28.881541 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:28.883178 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:28.884371 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:28.884982 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:28.885090 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:28.885872 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:28.885876 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:28.885877 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:10:28.886084 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:28.886133 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:10:28.886148 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:28.886218 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:28.886391 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 07:10:28.886465 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
I20260924 07:10:28.907792 9672 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:10:28.908353 9672 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:10:28.911736 9672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83707d3956b54931bd9a1731cb9ac5e0: Generated new TSK 0
I20260924 07:10:28.911938 9672 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:10:28.914433 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35561
--builtin_ntp_servers=127.0.98.20:37605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 07:10:28.996284 9686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:28.996484 9686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:28.996506 9686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:28.997717 9686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:28.997757 9686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:28.997781 9686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:10:28.999300 9686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:10:28.999547 9686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:28.999854 9686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:29.005651 9691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:29.005661 9694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:29.005661 9692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:29.005895 9686 server_base.cc:1034] running on GCE node
I20260924 07:10:29.006136 9686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:29.006394 9686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:29.007567 9686 hybrid_clock.cc:648] HybridClock initialized: now 1790233829007557 us; error 30 us; skew 500 ppm
I20260924 07:10:29.009012 9686 webserver.cc:460] Webserver started at http://127.0.98.1:37513/ using document root <none> and password file <none>
I20260924 07:10:29.009241 9686 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:29.009323 9686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:29.009461 9686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:29.010512 9686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-0/data/instance:
uuid: "42a8fee91c584157a96f21cdecb4dcb2"
format_stamp: "Formatted at 2026-09-24 07:10:29 on dist-test-slave-14ww"
server_key: "0cc6c0d2498e3d170efa992fcbfca10b"
server_key_iv: "8ec1397a4c244e5e8566a41ef03cb00a"
server_key_version: "encryptionkey@0"
I20260924 07:10:29.010874 9686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-0/wal/instance:
uuid: "42a8fee91c584157a96f21cdecb4dcb2"
format_stamp: "Formatted at 2026-09-24 07:10:29 on dist-test-slave-14ww"
server_key: "0cc6c0d2498e3d170efa992fcbfca10b"
server_key_iv: "8ec1397a4c244e5e8566a41ef03cb00a"
server_key_version: "encryptionkey@0"
I20260924 07:10:29.012168 9686 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:10:29.012989 9700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:29.013175 9686 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:10:29.013267 9686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-0/wal
uuid: "42a8fee91c584157a96f21cdecb4dcb2"
format_stamp: "Formatted at 2026-09-24 07:10:29 on dist-test-slave-14ww"
server_key: "0cc6c0d2498e3d170efa992fcbfca10b"
server_key_iv: "8ec1397a4c244e5e8566a41ef03cb00a"
server_key_version: "encryptionkey@0"
I20260924 07:10:29.013356 9686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:29.030128 9686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:29.030396 9686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:29.030539 9686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:29.030740 9686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:10:29.031042 9686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:10:29.031095 9686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:29.031127 9686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:10:29.031154 9686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:29.036475 9686 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33511
I20260924 07:10:29.036543 9813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33511 every 8 connection(s)
I20260924 07:10:29.036867 9686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:10:29.039574 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 9686
I20260924 07:10:29.039665 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:10:29.039849 392 external_mini_cluster.cc:1428] Setting key 26eceaf863a4173d24d0b305e1d68b21
I20260924 07:10:29.040818 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35561
--builtin_ntp_servers=127.0.98.20:37605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 07:10:29.042841 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:29.037564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50671 (local address 127.0.98.62:35561)
0924 07:10:29.037637 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:29.037639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:29.037977 (+ 338us) server_negotiation.cc:407] Connection header received
0924 07:10:29.038480 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:29.038482 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:29.038494 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:29.038559 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:29.039429 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:29.040186 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:29.041379 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:29.041603 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:29.041689 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:29.042560 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:29.042561 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:29.042561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:10:29.042566 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:29.042580 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:10:29.042584 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:29.042633 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:29.042734 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 07:10:29.042763 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 07:10:29.043079 9816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:29.037538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35561 (local address 127.0.98.1:50671)
0924 07:10:29.037853 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:29.037886 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:29.038360 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:29.038611 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:29.038614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:29.038622 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:29.039245 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:29.039251 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:29.040278 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:29.040279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:29.041287 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:29.041290 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:29.041702 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:29.041703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:29.041832 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:29.042414 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:10:29.042414 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:29.042443 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:10:29.042465 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:29.042645 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:29.042649 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:29.042652 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:10:29.042735 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 07:10:29.042832 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":106,"threads_started":1}
I20260924 07:10:29.043612 9814 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35561
I20260924 07:10:29.043702 9814 heartbeater.cc:464] Registering TS with master...
I20260924 07:10:29.043886 9814 heartbeater.cc:511] Master 127.0.98.62:35561 requested a full tablet report, sending...
I20260924 07:10:29.044353 9626 ts_manager.cc:194] Registered new tserver with Master: 42a8fee91c584157a96f21cdecb4dcb2 (127.0.98.1:33511)
I20260924 07:10:29.045054 9626 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:50671
W20260924 07:10:29.121083 9817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:29.121266 9817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:29.121286 9817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:29.122402 9817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:29.122439 9817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:29.122464 9817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:10:29.123740 9817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:10:29.123965 9817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:29.124239 9817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:29.129153 9822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:29.129189 9823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:29.129179 9825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:29.129370 9817 server_base.cc:1034] running on GCE node
I20260924 07:10:29.129575 9817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:29.129810 9817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:29.130972 9817 hybrid_clock.cc:648] HybridClock initialized: now 1790233829130966 us; error 39 us; skew 500 ppm
I20260924 07:10:29.132095 9817 webserver.cc:460] Webserver started at http://127.0.98.2:39551/ using document root <none> and password file <none>
I20260924 07:10:29.132332 9817 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:29.132398 9817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:29.132541 9817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:29.133615 9817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-1/data/instance:
uuid: "f1c154575e4b42789c3b48123efb542c"
format_stamp: "Formatted at 2026-09-24 07:10:29 on dist-test-slave-14ww"
server_key: "0d24b7dd3cabe93b92b4d0b5a3fee31e"
server_key_iv: "f5209281d877f37f77a64aa4e53f2e87"
server_key_version: "encryptionkey@0"
I20260924 07:10:29.133970 9817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f1c154575e4b42789c3b48123efb542c"
format_stamp: "Formatted at 2026-09-24 07:10:29 on dist-test-slave-14ww"
server_key: "0d24b7dd3cabe93b92b4d0b5a3fee31e"
server_key_iv: "f5209281d877f37f77a64aa4e53f2e87"
server_key_version: "encryptionkey@0"
I20260924 07:10:29.135224 9817 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:10:29.135947 9831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:29.136108 9817 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:10:29.136207 9817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-1/wal
uuid: "f1c154575e4b42789c3b48123efb542c"
format_stamp: "Formatted at 2026-09-24 07:10:29 on dist-test-slave-14ww"
server_key: "0d24b7dd3cabe93b92b4d0b5a3fee31e"
server_key_iv: "f5209281d877f37f77a64aa4e53f2e87"
server_key_version: "encryptionkey@0"
I20260924 07:10:29.136294 9817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:29.149413 9817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:29.149704 9817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:29.149852 9817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:29.150071 9817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:10:29.150393 9817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:10:29.150452 9817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:29.150493 9817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:10:29.150521 9817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:29.157886 9817 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35781
I20260924 07:10:29.157930 9944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35781 every 8 connection(s)
I20260924 07:10:29.158311 9817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:10:29.163735 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:29.159145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39123 (local address 127.0.98.62:35561)
0924 07:10:29.159217 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:29.159219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:29.159435 (+ 216us) server_negotiation.cc:407] Connection header received
0924 07:10:29.159988 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:29.159989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:29.160001 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:29.160065 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:29.160823 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:29.161321 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:29.162333 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:29.162534 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:29.162613 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:29.163441 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:29.163443 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:29.163443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:10:29.163448 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:29.163470 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:10:29.163473 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:29.163532 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:29.163625 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 07:10:29.163654 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 07:10:29.163950 9947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:29.159140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35561 (local address 127.0.98.2:39123)
0924 07:10:29.159316 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:29.159351 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:29.159877 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:29.160133 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:29.160137 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:29.160153 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:29.160635 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:29.160639 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:29.161420 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:29.161421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:29.162218 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:29.162219 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:29.162619 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:29.162620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:29.162754 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:29.163290 (+ 536us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:10:29.163290 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:29.163319 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:10:29.163340 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:29.163531 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:29.163537 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:29.163540 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:10:29.163627 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 07:10:29.163726 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":68,"threads_started":1}
I20260924 07:10:29.164469 9945 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35561
I20260924 07:10:29.164563 9945 heartbeater.cc:464] Registering TS with master...
I20260924 07:10:29.164700 9945 heartbeater.cc:511] Master 127.0.98.62:35561 requested a full tablet report, sending...
I20260924 07:10:29.165135 9626 ts_manager.cc:194] Registered new tserver with Master: f1c154575e4b42789c3b48123efb542c (127.0.98.2:35781)
I20260924 07:10:29.165488 9626 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:39123
I20260924 07:10:29.166226 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 9817
I20260924 07:10:29.166343 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:10:29.166558 392 external_mini_cluster.cc:1428] Setting key 270e9df71681c311b89efa9f89d4c934
I20260924 07:10:29.167469 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35561
--builtin_ntp_servers=127.0.98.20:37605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 07:10:29.247728 9948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:29.247920 9948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:29.247941 9948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:29.249181 9948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:29.249219 9948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:29.249244 9948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:10:29.250587 9948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:10:29.250800 9948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:29.251060 9948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:29.256353 9956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:29.256345 9953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:29.256501 9948 server_base.cc:1034] running on GCE node
W20260924 07:10:29.256346 9954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:29.256855 9948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:29.257088 9948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:29.258256 9948 hybrid_clock.cc:648] HybridClock initialized: now 1790233829258241 us; error 44 us; skew 500 ppm
I20260924 07:10:29.259479 9948 webserver.cc:460] Webserver started at http://127.0.98.3:35057/ using document root <none> and password file <none>
I20260924 07:10:29.259721 9948 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:29.259790 9948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:29.259924 9948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:29.261029 9948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-2/data/instance:
uuid: "a26cfb973b3f4d2e8329523c7c2b44c0"
format_stamp: "Formatted at 2026-09-24 07:10:29 on dist-test-slave-14ww"
server_key: "c04668828bf4b02704098258be53c83d"
server_key_iv: "128402642b21e0212303a75fb5cb43c9"
server_key_version: "encryptionkey@0"
I20260924 07:10:29.261381 9948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a26cfb973b3f4d2e8329523c7c2b44c0"
format_stamp: "Formatted at 2026-09-24 07:10:29 on dist-test-slave-14ww"
server_key: "c04668828bf4b02704098258be53c83d"
server_key_iv: "128402642b21e0212303a75fb5cb43c9"
server_key_version: "encryptionkey@0"
I20260924 07:10:29.262598 9948 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:10:29.263262 9962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:29.263456 9948 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:10:29.263550 9948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-2/wal
uuid: "a26cfb973b3f4d2e8329523c7c2b44c0"
format_stamp: "Formatted at 2026-09-24 07:10:29 on dist-test-slave-14ww"
server_key: "c04668828bf4b02704098258be53c83d"
server_key_iv: "128402642b21e0212303a75fb5cb43c9"
server_key_version: "encryptionkey@0"
I20260924 07:10:29.263633 9948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:29.274361 9948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:29.274653 9948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:29.274806 9948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:29.275015 9948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:10:29.275331 9948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:10:29.275390 9948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:29.275434 9948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:10:29.275460 9948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:29.281144 9948 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37207
I20260924 07:10:29.281190 10075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37207 every 8 connection(s)
I20260924 07:10:29.281484 9948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:10:29.281939 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 9948
I20260924 07:10:29.282045 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790233772407728-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:10:29.282312 392 external_mini_cluster.cc:1428] Setting key ea6c42a8a1de9a0d2e23a8729479e217
I20260924 07:10:29.286692 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:29.282174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41367 (local address 127.0.98.62:35561)
0924 07:10:29.282239 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:29.282242 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:29.282572 (+ 330us) server_negotiation.cc:407] Connection header received
0924 07:10:29.283223 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:29.283224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:29.283238 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:29.283297 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:29.284070 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:29.284546 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:29.285535 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:29.285714 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:29.285769 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:29.286484 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:29.286485 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:29.286485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:10:29.286489 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:29.286500 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:10:29.286504 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:29.286554 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:29.286615 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 07:10:29.286636 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 07:10:29.286947 10078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:29.282174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35561 (local address 127.0.98.3:41367)
0924 07:10:29.282381 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:29.282424 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:29.283126 (+ 702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:29.283372 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:29.283377 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:29.283387 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:29.283995 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:29.284000 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:29.284629 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:29.284630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:29.285459 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:29.285461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:29.285818 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:29.285818 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:29.285921 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:29.286362 (+ 441us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:10:29.286362 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:29.286388 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:10:29.286408 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:29.286556 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:29.286558 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:29.286560 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 07:10:29.286601 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 07:10:29.286671 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":99,"threads_started":1}
I20260924 07:10:29.287775 10076 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35561
I20260924 07:10:29.287878 10076 heartbeater.cc:464] Registering TS with master...
I20260924 07:10:29.288121 10076 heartbeater.cc:511] Master 127.0.98.62:35561 requested a full tablet report, sending...
I20260924 07:10:29.288491 9626 ts_manager.cc:194] Registered new tserver with Master: a26cfb973b3f4d2e8329523c7c2b44c0 (127.0.98.3:37207)
I20260924 07:10:29.288949 9626 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:41367
I20260924 07:10:29.294116 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 07:10:29.296037 9666 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:29.295465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60238 (local address 127.0.98.62:35561)
0924 07:10:29.295543 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:29.295546 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:29.295654 (+ 108us) server_negotiation.cc:407] Connection header received
0924 07:10:29.295715 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:29.295716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:29.295728 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:29.295798 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:29.295977 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60238: BlockingRecv error: recv got EOF from 127.0.0.1:60238 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 07:10:29.296096 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 9686
I20260924 07:10:29.300841 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 9817
I20260924 07:10:29.305516 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 9948
I20260924 07:10:29.309772 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 9595
2026-09-24T07:10:29Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (580 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T07:10:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:10:29Z Disabled control of system clock
I20260924 07:10:29.335752 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44321
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 07:10:29.416023 10087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:29.416251 10087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:29.416287 10087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:29.417521 10087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:10:29.417580 10087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:29.417604 10087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:29.417634 10087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:10:29.417663 10087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:10:29.418988 10087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44321
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:10:29.419255 10087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:29.419534 10087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:29.424551 10093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:29.424551 10095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:29.424620 10092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:29.425019 10087 server_base.cc:1034] running on GCE node
I20260924 07:10:29.425251 10087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:29.425498 10087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:29.426637 10087 hybrid_clock.cc:648] HybridClock initialized: now 1790233829426624 us; error 33 us; skew 500 ppm
I20260924 07:10:29.427901 10087 webserver.cc:460] Webserver started at http://127.0.98.62:39001/ using document root <none> and password file <none>
I20260924 07:10:29.428167 10087 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:29.428256 10087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:29.428406 10087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:29.429584 10087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "d02bc802ffd64c77b2d9b1b107c6da6b"
format_stamp: "Formatted at 2026-09-24 07:10:29 on dist-test-slave-14ww"
server_key: "3197b0d25b85a010e66984bde4b11faf"
server_key_iv: "c8ef4b185de4dc4b57e4e3a9ffafe11f"
server_key_version: "encryptionkey@0"
I20260924 07:10:29.429955 10087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "d02bc802ffd64c77b2d9b1b107c6da6b"
format_stamp: "Formatted at 2026-09-24 07:10:29 on dist-test-slave-14ww"
server_key: "3197b0d25b85a010e66984bde4b11faf"
server_key_iv: "c8ef4b185de4dc4b57e4e3a9ffafe11f"
server_key_version: "encryptionkey@0"
I20260924 07:10:29.431221 10087 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 07:10:29.431890 10101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:29.432080 10087 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:10:29.432179 10087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "d02bc802ffd64c77b2d9b1b107c6da6b"
format_stamp: "Formatted at 2026-09-24 07:10:29 on dist-test-slave-14ww"
server_key: "3197b0d25b85a010e66984bde4b11faf"
server_key_iv: "c8ef4b185de4dc4b57e4e3a9ffafe11f"
server_key_version: "encryptionkey@0"
I20260924 07:10:29.432251 10087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:29.435590 10087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:29.435830 10087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:29.435981 10087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:29.439743 10087 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44321
I20260924 07:10:29.439797 10153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44321 every 8 connection(s)
I20260924 07:10:29.440129 10087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:10:29.440874 10154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:29.443526 10154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d02bc802ffd64c77b2d9b1b107c6da6b: Bootstrap starting.
I20260924 07:10:29.444123 10154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d02bc802ffd64c77b2d9b1b107c6da6b: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:29.444409 10154 log.cc:824] T 00000000000000000000000000000000 P d02bc802ffd64c77b2d9b1b107c6da6b: Log is configured to *not* fsync() on all Append() calls
I20260924 07:10:29.445160 10154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d02bc802ffd64c77b2d9b1b107c6da6b: No bootstrap required, opened a new log
I20260924 07:10:29.446779 10154 raft_consensus.cc:348] T 00000000000000000000000000000000 P d02bc802ffd64c77b2d9b1b107c6da6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02bc802ffd64c77b2d9b1b107c6da6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44321 } }
I20260924 07:10:29.446913 10154 raft_consensus.cc:374] T 00000000000000000000000000000000 P d02bc802ffd64c77b2d9b1b107c6da6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:29.446935 10154 raft_consensus.cc:729] T 00000000000000000000000000000000 P d02bc802ffd64c77b2d9b1b107c6da6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d02bc802ffd64c77b2d9b1b107c6da6b, State: Initialized, Role: FOLLOWER
I20260924 07:10:29.447081 10154 consensus_queue.cc:260] T 00000000000000000000000000000000 P d02bc802ffd64c77b2d9b1b107c6da6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02bc802ffd64c77b2d9b1b107c6da6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44321 } }
I20260924 07:10:29.447155 10154 raft_consensus.cc:388] T 00000000000000000000000000000000 P d02bc802ffd64c77b2d9b1b107c6da6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:10:29.447209 10154 raft_consensus.cc:482] T 00000000000000000000000000000000 P d02bc802ffd64c77b2d9b1b107c6da6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:10:29.447255 10154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d02bc802ffd64c77b2d9b1b107c6da6b [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:29.447801 10154 raft_consensus.cc:504] T 00000000000000000000000000000000 P d02bc802ffd64c77b2d9b1b107c6da6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02bc802ffd64c77b2d9b1b107c6da6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44321 } }
I20260924 07:10:29.447885 10154 leader_election.cc:302] T 00000000000000000000000000000000 P d02bc802ffd64c77b2d9b1b107c6da6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d02bc802ffd64c77b2d9b1b107c6da6b; no voters:
I20260924 07:10:29.448060 10154 leader_election.cc:288] T 00000000000000000000000000000000 P d02bc802ffd64c77b2d9b1b107c6da6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:10:29.448171 10157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d02bc802ffd64c77b2d9b1b107c6da6b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:10:29.448307 10154 sys_catalog.cc:564] T 00000000000000000000000000000000 P d02bc802ffd64c77b2d9b1b107c6da6b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:10:29.448351 10157 raft_consensus.cc:686] T 00000000000000000000000000000000 P d02bc802ffd64c77b2d9b1b107c6da6b [term 1 LEADER]: Becoming Leader. State: Replica: d02bc802ffd64c77b2d9b1b107c6da6b, State: Running, Role: LEADER
I20260924 07:10:29.448442 10157 consensus_queue.cc:237] T 00000000000000000000000000000000 P d02bc802ffd64c77b2d9b1b107c6da6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02bc802ffd64c77b2d9b1b107c6da6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44321 } }
I20260924 07:10:29.448917 10158 sys_catalog.cc:455] T 00000000000000000000000000000000 P d02bc802ffd64c77b2d9b1b107c6da6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d02bc802ffd64c77b2d9b1b107c6da6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02bc802ffd64c77b2d9b1b107c6da6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44321 } } }
I20260924 07:10:29.449005 10158 sys_catalog.cc:458] T 00000000000000000000000000000000 P d02bc802ffd64c77b2d9b1b107c6da6b [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:29.449160 10159 sys_catalog.cc:455] T 00000000000000000000000000000000 P d02bc802ffd64c77b2d9b1b107c6da6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d02bc802ffd64c77b2d9b1b107c6da6b. Latest consensus state: current_term: 1 leader_uuid: "d02bc802ffd64c77b2d9b1b107c6da6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02bc802ffd64c77b2d9b1b107c6da6b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44321 } } }
I20260924 07:10:29.449223 10159 sys_catalog.cc:458] T 00000000000000000000000000000000 P d02bc802ffd64c77b2d9b1b107c6da6b [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:29.449550 10166 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:10:29.449690 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 10087
I20260924 07:10:29.449795 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:10:29.449994 392 external_mini_cluster.cc:1428] Setting key 1bbd9af871af8a3acc43ae97ce9b3585
I20260924 07:10:29.450223 10166 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:10:29.451723 10166 catalog_manager.cc:1347] Generated new cluster ID: f141109f74cd4c75874f26c89cf1869b
I20260924 07:10:29.451774 10166 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:10:29.452804 10176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:29.450607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36486 (local address 127.0.98.62:44321)
0924 07:10:29.450807 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:29.450816 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:29.450847 (+ 31us) server_negotiation.cc:407] Connection header received
0924 07:10:29.450967 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:29.450975 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:29.451023 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:29.451158 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:29.452199 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:29.452203 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:29.452203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:10:29.452208 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:29.452254 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:10:29.452271 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:29.452324 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:29.452544 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 07:10:29.452598 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":102,"threads_started":1}
I20260924 07:10:29.473599 10166 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:10:29.473804 10166 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:10:29.483664 10166 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d02bc802ffd64c77b2d9b1b107c6da6b: Generated new TSK 0
I20260924 07:10:29.483860 10166 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:10:29.490659 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44321
--builtin_ntp_servers=127.0.98.20:34117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 07:10:29.570152 10178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:29.570340 10178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:29.570361 10178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:29.571497 10178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:29.571537 10178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:29.571563 10178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:10:29.572894 10178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:10:29.573130 10178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:29.573413 10178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:29.578325 10184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:29.578333 10183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:29.578408 10186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:29.578361 10178 server_base.cc:1034] running on GCE node
I20260924 07:10:29.578696 10178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:29.578974 10178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:29.580147 10178 hybrid_clock.cc:648] HybridClock initialized: now 1790233829580142 us; error 37 us; skew 500 ppm
I20260924 07:10:29.581437 10178 webserver.cc:460] Webserver started at http://127.0.98.1:32947/ using document root <none> and password file <none>
I20260924 07:10:29.581676 10178 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:29.581745 10178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:29.581880 10178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:29.582931 10178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-0/data/instance:
uuid: "aa5f654ca86b453299222d3bda041c6d"
format_stamp: "Formatted at 2026-09-24 07:10:29 on dist-test-slave-14ww"
server_key: "a5a8f53827e04ad2370ad1a4b95b2af5"
server_key_iv: "053163f858954fbbd0553f97a1c604a6"
server_key_version: "encryptionkey@0"
I20260924 07:10:29.583289 10178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-0/wal/instance:
uuid: "aa5f654ca86b453299222d3bda041c6d"
format_stamp: "Formatted at 2026-09-24 07:10:29 on dist-test-slave-14ww"
server_key: "a5a8f53827e04ad2370ad1a4b95b2af5"
server_key_iv: "053163f858954fbbd0553f97a1c604a6"
server_key_version: "encryptionkey@0"
I20260924 07:10:29.584564 10178 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:10:29.585408 10192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:29.585567 10178 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:10:29.585664 10178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-0/wal
uuid: "aa5f654ca86b453299222d3bda041c6d"
format_stamp: "Formatted at 2026-09-24 07:10:29 on dist-test-slave-14ww"
server_key: "a5a8f53827e04ad2370ad1a4b95b2af5"
server_key_iv: "053163f858954fbbd0553f97a1c604a6"
server_key_version: "encryptionkey@0"
I20260924 07:10:29.585749 10178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:29.588968 10178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:29.589210 10178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:29.589345 10178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:29.589556 10178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:10:29.589838 10178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:10:29.589897 10178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:29.589927 10178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:10:29.589955 10178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:29.595546 10178 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44235
I20260924 07:10:29.595597 10305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44235 every 8 connection(s)
I20260924 07:10:29.595885 10178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:10:29.598626 10176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:29.596541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56389 (local address 127.0.98.62:44321)
0924 07:10:29.596596 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:29.596598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:29.596814 (+ 216us) server_negotiation.cc:407] Connection header received
0924 07:10:29.597222 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:29.597223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:29.597229 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:29.597296 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:29.598402 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:29.598403 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:29.598403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:10:29.598407 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:29.598422 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:10:29.598425 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:29.598478 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:29.598561 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 07:10:29.598575 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 07:10:29.598807 10308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:29.596510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44321 (local address 127.0.98.1:56389)
0924 07:10:29.596673 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:29.596710 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:29.597141 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:29.597345 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:29.597348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:29.597356 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:29.598237 (+ 881us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:10:29.598238 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:29.598264 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:10:29.598290 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:29.598491 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:29.598495 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:29.598498 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:10:29.598555 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 07:10:29.598601 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":79,"threads_started":1}
I20260924 07:10:29.599292 10306 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44321
I20260924 07:10:29.599396 10306 heartbeater.cc:464] Registering TS with master...
I20260924 07:10:29.599575 10306 heartbeater.cc:511] Master 127.0.98.62:44321 requested a full tablet report, sending...
I20260924 07:10:29.600122 10118 ts_manager.cc:194] Registered new tserver with Master: aa5f654ca86b453299222d3bda041c6d (127.0.98.1:44235)
I20260924 07:10:29.605175 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 10178
I20260924 07:10:29.605254 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:10:29.605400 392 external_mini_cluster.cc:1428] Setting key 8f82df120dca60f81d20fb8e937100df
I20260924 07:10:29.606309 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44321
--builtin_ntp_servers=127.0.98.20:34117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 07:10:29.685258 10309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:29.685443 10309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:29.685465 10309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:29.686614 10309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:29.686651 10309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:29.686676 10309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:10:29.688007 10309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:10:29.688292 10309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:29.688551 10309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:29.693706 10314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:29.693714 10315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:29.693711 10317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:29.694022 10309 server_base.cc:1034] running on GCE node
I20260924 07:10:29.694224 10309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:29.694434 10309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:29.695629 10309 hybrid_clock.cc:648] HybridClock initialized: now 1790233829695612 us; error 34 us; skew 500 ppm
I20260924 07:10:29.696920 10309 webserver.cc:460] Webserver started at http://127.0.98.2:38527/ using document root <none> and password file <none>
I20260924 07:10:29.697120 10309 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:29.697161 10309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:29.697250 10309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:29.698294 10309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-1/data/instance:
uuid: "605026c15d82467abb46e1ab3a258103"
format_stamp: "Formatted at 2026-09-24 07:10:29 on dist-test-slave-14ww"
server_key: "f12438b603a2e1fa4464f557a3fe778f"
server_key_iv: "93e267d7c37fd6668f413b7a9354d893"
server_key_version: "encryptionkey@0"
I20260924 07:10:29.698582 10309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-1/wal/instance:
uuid: "605026c15d82467abb46e1ab3a258103"
format_stamp: "Formatted at 2026-09-24 07:10:29 on dist-test-slave-14ww"
server_key: "f12438b603a2e1fa4464f557a3fe778f"
server_key_iv: "93e267d7c37fd6668f413b7a9354d893"
server_key_version: "encryptionkey@0"
I20260924 07:10:29.699744 10309 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:10:29.700428 10323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:29.700593 10309 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:10:29.700692 10309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-1/wal
uuid: "605026c15d82467abb46e1ab3a258103"
format_stamp: "Formatted at 2026-09-24 07:10:29 on dist-test-slave-14ww"
server_key: "f12438b603a2e1fa4464f557a3fe778f"
server_key_iv: "93e267d7c37fd6668f413b7a9354d893"
server_key_version: "encryptionkey@0"
I20260924 07:10:29.700791 10309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:29.703958 10309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:29.704202 10309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:29.704336 10309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:29.704553 10309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:10:29.704905 10309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:10:29.704958 10309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:29.704996 10309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:10:29.705022 10309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:29.710673 10309 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36647
I20260924 07:10:29.710700 10436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36647 every 8 connection(s)
I20260924 07:10:29.711014 10309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:10:29.713680 10176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:29.711697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50933 (local address 127.0.98.62:44321)
0924 07:10:29.711741 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:29.711743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:29.711961 (+ 218us) server_negotiation.cc:407] Connection header received
0924 07:10:29.712367 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:29.712368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:29.712373 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:29.712427 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:29.713498 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:29.713499 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:29.713499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:10:29.713502 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:29.713516 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:10:29.713520 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:29.713574 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:29.713618 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 07:10:29.713629 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 07:10:29.713898 10439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:29.711660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44321 (local address 127.0.98.2:50933)
0924 07:10:29.711837 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:29.711874 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:29.712291 (+ 417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:29.712477 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:29.712480 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:29.712487 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:29.713353 (+ 866us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:10:29.713354 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:29.713387 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:10:29.713412 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:29.713572 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:29.713574 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:29.713576 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 07:10:29.713625 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 07:10:29.713668 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":93,"threads_started":1}
I20260924 07:10:29.714424 10437 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44321
I20260924 07:10:29.714540 10437 heartbeater.cc:464] Registering TS with master...
I20260924 07:10:29.714692 10437 heartbeater.cc:511] Master 127.0.98.62:44321 requested a full tablet report, sending...
I20260924 07:10:29.715090 10118 ts_manager.cc:194] Registered new tserver with Master: 605026c15d82467abb46e1ab3a258103 (127.0.98.2:36647)
I20260924 07:10:29.721007 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 10309
I20260924 07:10:29.721102 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:10:29.721253 392 external_mini_cluster.cc:1428] Setting key db0e129c2988cbd06e4edf7d89d45da5
I20260924 07:10:29.722096 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44321
--builtin_ntp_servers=127.0.98.20:34117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 07:10:29.800866 10440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:29.801056 10440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:29.801077 10440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:29.802170 10440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:29.802202 10440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:29.802227 10440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:10:29.803534 10440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:10:29.803779 10440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:29.804083 10440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:29.808972 10448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:29.808961 10445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:29.808961 10446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:29.809126 10440 server_base.cc:1034] running on GCE node
I20260924 07:10:29.809417 10440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:29.809653 10440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:29.810793 10440 hybrid_clock.cc:648] HybridClock initialized: now 1790233829810781 us; error 28 us; skew 500 ppm
I20260924 07:10:29.811931 10440 webserver.cc:460] Webserver started at http://127.0.98.3:37897/ using document root <none> and password file <none>
I20260924 07:10:29.812193 10440 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:29.812237 10440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:29.812384 10440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:29.813495 10440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-2/data/instance:
uuid: "3887ef5819f84ada87cdc0c28d7c8d57"
format_stamp: "Formatted at 2026-09-24 07:10:29 on dist-test-slave-14ww"
server_key: "ab6a6c4afbd469d0b950690ace2d4f03"
server_key_iv: "1c11ddd3d59dea5657bbfa496db976d9"
server_key_version: "encryptionkey@0"
I20260924 07:10:29.813848 10440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3887ef5819f84ada87cdc0c28d7c8d57"
format_stamp: "Formatted at 2026-09-24 07:10:29 on dist-test-slave-14ww"
server_key: "ab6a6c4afbd469d0b950690ace2d4f03"
server_key_iv: "1c11ddd3d59dea5657bbfa496db976d9"
server_key_version: "encryptionkey@0"
I20260924 07:10:29.815119 10440 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:10:29.815799 10454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:29.815976 10440 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:10:29.816079 10440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-2/wal
uuid: "3887ef5819f84ada87cdc0c28d7c8d57"
format_stamp: "Formatted at 2026-09-24 07:10:29 on dist-test-slave-14ww"
server_key: "ab6a6c4afbd469d0b950690ace2d4f03"
server_key_iv: "1c11ddd3d59dea5657bbfa496db976d9"
server_key_version: "encryptionkey@0"
I20260924 07:10:29.816164 10440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:29.819425 10440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:29.819664 10440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:29.819798 10440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:29.820001 10440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:10:29.820297 10440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:10:29.820355 10440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:29.820394 10440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:10:29.820420 10440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:29.826562 10440 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33793
I20260924 07:10:29.826594 10567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33793 every 8 connection(s)
I20260924 07:10:29.826886 10440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:10:29.829876 10176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:29.827581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47973 (local address 127.0.98.62:44321)
0924 07:10:29.827639 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:29.827641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:29.827811 (+ 170us) server_negotiation.cc:407] Connection header received
0924 07:10:29.828317 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:29.828319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:29.828326 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:29.828376 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:29.829595 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:29.829597 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:29.829597 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:10:29.829602 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:29.829617 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:10:29.829622 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:29.829697 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:29.829790 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 07:10:29.829807 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 07:10:29.830075 10570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:29.827518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44321 (local address 127.0.98.3:47973)
0924 07:10:29.827703 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:29.827733 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:29.828197 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:29.828442 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:29.828445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:29.828457 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:29.829439 (+ 982us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:10:29.829439 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:29.829468 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:10:29.829493 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:29.829697 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:29.829703 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:29.829706 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:10:29.829798 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 07:10:29.829856 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
I20260924 07:10:29.830663 10568 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44321
I20260924 07:10:29.830751 10568 heartbeater.cc:464] Registering TS with master...
I20260924 07:10:29.830850 10568 heartbeater.cc:511] Master 127.0.98.62:44321 requested a full tablet report, sending...
I20260924 07:10:29.831158 10118 ts_manager.cc:194] Registered new tserver with Master: 3887ef5819f84ada87cdc0c28d7c8d57 (127.0.98.3:33793)
I20260924 07:10:29.836314 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 10440
I20260924 07:10:29.836431 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790233772407728-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:10:29.836642 392 external_mini_cluster.cc:1428] Setting key 81404660d1fe43fa937a4320e4076529
I20260924 07:10:29.837457 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 07:10:29.839131 10176 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:10:29.838735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36492 (local address 127.0.98.62:44321)
0924 07:10:29.838799 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:29.838801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:29.838893 (+ 92us) server_negotiation.cc:407] Connection header received
0924 07:10:29.838965 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:29.838966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:29.838969 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:29.839008 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:29.839092 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36492: BlockingRecv error: recv got EOF from 127.0.0.1:36492 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 07:10:29.839246 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 10178
I20260924 07:10:29.843966 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 10309
I20260924 07:10:29.848011 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 10440
I20260924 07:10:29.852043 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 10087
2026-09-24T07:10:29Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (542 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:10:29 dist-test-slave-14ww krb5kdc[10578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:10:29 dist-test-slave-14ww krb5kdc[10578](info): set up 2 sockets
Sep 24 07:10:29 dist-test-slave-14ww krb5kdc[10578](info): commencing operation
krb5kdc: starting...
W20260924 07:10:30.889734 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:10:30 dist-test-slave-14ww krb5kdc[10578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:10:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:10:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:10:31.020627 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44805
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44805
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:31.100014 10591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:31.100207 10591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:31.100229 10591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:31.101379 10591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:10:31.101420 10591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:31.101434 10591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:31.101472 10591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:10:31.101503 10591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:10:31.102917 10591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:10:31.103168 10591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:31.103405 10591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:31.108120 10599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:31.108134 10596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:31.108196 10597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:31.108289 10591 server_base.cc:1034] running on GCE node
I20260924 07:10:31.108662 10591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:31.108959 10591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:31.110126 10591 hybrid_clock.cc:648] HybridClock initialized: now 1790233831110121 us; error 33 us; skew 500 ppm
Sep 24 07:10:31 dist-test-slave-14ww krb5kdc[10578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:31.112416 10591 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:10:31.112941 10591 webserver.cc:460] Webserver started at http://127.0.98.62:39553/ using document root <none> and password file <none>
I20260924 07:10:31.113161 10591 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:31.113206 10591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:31.113332 10591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:31.114310 10591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "ba5bb29d44e54b0a8ec12298899dbe34"
format_stamp: "Formatted at 2026-09-24 07:10:31 on dist-test-slave-14ww"
server_key: "ac8a4094eb539a2538b41a449236657b"
server_key_iv: "8019a9719267e4afa6e9209efa00d840"
server_key_version: "encryptionkey@0"
I20260924 07:10:31.114679 10591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "ba5bb29d44e54b0a8ec12298899dbe34"
format_stamp: "Formatted at 2026-09-24 07:10:31 on dist-test-slave-14ww"
server_key: "ac8a4094eb539a2538b41a449236657b"
server_key_iv: "8019a9719267e4afa6e9209efa00d840"
server_key_version: "encryptionkey@0"
I20260924 07:10:31.115934 10591 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:10:31.116591 10606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:31.116806 10591 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:10:31.116901 10591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "ba5bb29d44e54b0a8ec12298899dbe34"
format_stamp: "Formatted at 2026-09-24 07:10:31 on dist-test-slave-14ww"
server_key: "ac8a4094eb539a2538b41a449236657b"
server_key_iv: "8019a9719267e4afa6e9209efa00d840"
server_key_version: "encryptionkey@0"
I20260924 07:10:31.116997 10591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:31.137229 10591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:31.139334 10591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:31.139503 10591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:31.143735 10591 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44805
I20260924 07:10:31.143774 10658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44805 every 8 connection(s)
I20260924 07:10:31.144124 10591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:10:31.145151 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 10591
I20260924 07:10:31.145238 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:10:31.145226 10659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:31.145555 392 external_mini_cluster.cc:1428] Setting key 86a06abec179b00f129e306eb81c4f51
I20260924 07:10:31.148622 10659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba5bb29d44e54b0a8ec12298899dbe34: Bootstrap starting.
I20260924 07:10:31.149333 10659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba5bb29d44e54b0a8ec12298899dbe34: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:31.149626 10659 log.cc:824] T 00000000000000000000000000000000 P ba5bb29d44e54b0a8ec12298899dbe34: Log is configured to *not* fsync() on all Append() calls
I20260924 07:10:31.150352 10659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba5bb29d44e54b0a8ec12298899dbe34: No bootstrap required, opened a new log
I20260924 07:10:31.152137 10659 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba5bb29d44e54b0a8ec12298899dbe34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5bb29d44e54b0a8ec12298899dbe34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44805 } }
I20260924 07:10:31.152297 10659 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba5bb29d44e54b0a8ec12298899dbe34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:31.152345 10659 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba5bb29d44e54b0a8ec12298899dbe34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba5bb29d44e54b0a8ec12298899dbe34, State: Initialized, Role: FOLLOWER
Sep 24 07:10:31 dist-test-slave-14ww krb5kdc[10578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:10:31.152498 10659 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba5bb29d44e54b0a8ec12298899dbe34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5bb29d44e54b0a8ec12298899dbe34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44805 } }
I20260924 07:10:31.152604 10659 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba5bb29d44e54b0a8ec12298899dbe34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:10:31.152652 10659 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba5bb29d44e54b0a8ec12298899dbe34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:10:31.152698 10659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba5bb29d44e54b0a8ec12298899dbe34 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:31.153385 10659 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba5bb29d44e54b0a8ec12298899dbe34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5bb29d44e54b0a8ec12298899dbe34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44805 } }
I20260924 07:10:31.153509 10659 leader_election.cc:302] T 00000000000000000000000000000000 P ba5bb29d44e54b0a8ec12298899dbe34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba5bb29d44e54b0a8ec12298899dbe34; no voters:
I20260924 07:10:31.153707 10659 leader_election.cc:288] T 00000000000000000000000000000000 P ba5bb29d44e54b0a8ec12298899dbe34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:10:31.153784 10664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba5bb29d44e54b0a8ec12298899dbe34 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:10:31.153959 10664 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba5bb29d44e54b0a8ec12298899dbe34 [term 1 LEADER]: Becoming Leader. State: Replica: ba5bb29d44e54b0a8ec12298899dbe34, State: Running, Role: LEADER
I20260924 07:10:31.153976 10659 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba5bb29d44e54b0a8ec12298899dbe34 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:10:31.154052 10664 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba5bb29d44e54b0a8ec12298899dbe34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5bb29d44e54b0a8ec12298899dbe34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44805 } }
I20260924 07:10:31.154408 10666 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba5bb29d44e54b0a8ec12298899dbe34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba5bb29d44e54b0a8ec12298899dbe34. Latest consensus state: current_term: 1 leader_uuid: "ba5bb29d44e54b0a8ec12298899dbe34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5bb29d44e54b0a8ec12298899dbe34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44805 } } }
I20260924 07:10:31.154603 10666 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba5bb29d44e54b0a8ec12298899dbe34 [sys.catalog]: This master's current role is: LEADER
I20260924 07:10:31.154417 10665 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba5bb29d44e54b0a8ec12298899dbe34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba5bb29d44e54b0a8ec12298899dbe34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba5bb29d44e54b0a8ec12298899dbe34" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44805 } } }
I20260924 07:10:31.155717 10665 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba5bb29d44e54b0a8ec12298899dbe34 [sys.catalog]: This master's current role is: LEADER
W20260924 07:10:31.155781 10680 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ba5bb29d44e54b0a8ec12298899dbe34: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:10:31.155824 10680 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:10:31.155915 10673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:10:31.156083 10673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:10:31.157137 10662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:31.146138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53870 (local address 127.0.98.62:44805)
0924 07:10:31.146334 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:31.146344 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:31.146376 (+ 32us) server_negotiation.cc:407] Connection header received
0924 07:10:31.146853 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:31.146862 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:31.146927 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:31.147085 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:31.147687 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:31.149077 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:31.149923 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:31.150415 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:31.150522 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:31.153155 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:31.153159 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:31.153160 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:31.153169 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:31.155552 (+ 2383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:31.155958 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:31.155962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:31.155962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:31.156069 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:31.156281 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:31.156282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:31.156282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:31.156513 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:31.156676 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:31.156809 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 07:10:31.156945 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":78,"threads_started":1}
I20260924 07:10:31.161077 10673 catalog_manager.cc:1347] Generated new cluster ID: 19f8d51ae56846ba8ccbecd7252735ac
I20260924 07:10:31.161130 10673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:10:31.202486 10673 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:10:31.203227 10673 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:10:31.208189 10673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba5bb29d44e54b0a8ec12298899dbe34: Generated new TSK 0
I20260924 07:10:31.208360 10673 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 07:10:31.246635 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44805
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:31.326309 10685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:31.326504 10685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:31.326527 10685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:31.327584 10685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:31.327617 10685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:31.327642 10685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:10:31.329056 10685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:10:31.329228 10685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:31.329391 10685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:31.334251 10691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:31.334324 10693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:31.334318 10690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:31.334605 10685 server_base.cc:1034] running on GCE node
I20260924 07:10:31.334803 10685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:31.335074 10685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:31.336244 10685 hybrid_clock.cc:648] HybridClock initialized: now 1790233831336234 us; error 29 us; skew 500 ppm
Sep 24 07:10:31 dist-test-slave-14ww krb5kdc[10578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:31.338728 10685 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:10:31.339109 10685 webserver.cc:460] Webserver started at http://127.0.98.1:35419/ using document root <none> and password file <none>
I20260924 07:10:31.339330 10685 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:31.339377 10685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:31.339514 10685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:31.340535 10685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-0/data/instance:
uuid: "53a69ae0569e438fa37419b55212629c"
format_stamp: "Formatted at 2026-09-24 07:10:31 on dist-test-slave-14ww"
server_key: "c3a40cf5f4b3ab78b2dce73edf82c90a"
server_key_iv: "34e14119a8c6ad6311f0263334453621"
server_key_version: "encryptionkey@0"
I20260924 07:10:31.340912 10685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-0/wal/instance:
uuid: "53a69ae0569e438fa37419b55212629c"
format_stamp: "Formatted at 2026-09-24 07:10:31 on dist-test-slave-14ww"
server_key: "c3a40cf5f4b3ab78b2dce73edf82c90a"
server_key_iv: "34e14119a8c6ad6311f0263334453621"
server_key_version: "encryptionkey@0"
I20260924 07:10:31.342187 10685 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:10:31.342932 10700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:31.343122 10685 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:10:31.343217 10685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-0/wal
uuid: "53a69ae0569e438fa37419b55212629c"
format_stamp: "Formatted at 2026-09-24 07:10:31 on dist-test-slave-14ww"
server_key: "c3a40cf5f4b3ab78b2dce73edf82c90a"
server_key_iv: "34e14119a8c6ad6311f0263334453621"
server_key_version: "encryptionkey@0"
I20260924 07:10:31.343297 10685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:31.353010 10685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:31.355394 10685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:31.355544 10685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:31.355778 10685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:10:31.356076 10685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:10:31.356127 10685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:31.356168 10685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:10:31.356194 10685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:31.361917 10685 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33213
I20260924 07:10:31.361954 10813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33213 every 8 connection(s)
I20260924 07:10:31.362246 10685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 07:10:31 dist-test-slave-14ww krb5kdc[10578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:10:31.370723 10662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:31.362962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41791 (local address 127.0.98.62:44805)
0924 07:10:31.363038 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:31.363040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:31.363252 (+ 212us) server_negotiation.cc:407] Connection header received
0924 07:10:31.363671 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:31.363672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:31.363682 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:31.363733 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:31.364552 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:31.365124 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:31.366060 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:31.366250 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:31.366301 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:31.368239 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:31.368240 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:31.368240 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:31.368245 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:31.369553 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:31.369901 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:31.369901 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:31.369902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:31.369949 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:31.370206 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:31.370206 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:31.370207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:31.370298 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:31.370347 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:31.370603 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 07:10:31.370660 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 07:10:31.370894 10816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:31.362946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44805 (local address 127.0.98.1:41791)
0924 07:10:31.363138 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:31.363171 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:31.363594 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:31.363779 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:31.363782 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:31.364303 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:31.364481 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:31.364485 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:31.365197 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:31.365198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:31.365985 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:31.365987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:31.366318 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:31.366319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:31.366408 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:31.366912 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:31.366919 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:31.368163 (+ 1244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:31.369629 (+ 1466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:31.369634 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:31.369635 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:31.369830 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:31.370027 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:31.370028 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:31.370028 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:31.370133 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:31.370374 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:31.370379 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:31.370484 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 07:10:31.370577 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 07:10:31.370693 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":117,"threads_started":1}
I20260924 07:10:31.371140 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 10685
I20260924 07:10:31.371266 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:10:31.371416 10814 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44805
I20260924 07:10:31.371477 392 external_mini_cluster.cc:1428] Setting key e98e26dfde99815298f6cd14f5a8e320
I20260924 07:10:31.371517 10814 heartbeater.cc:464] Registering TS with master...
I20260924 07:10:31.371682 10814 heartbeater.cc:511] Master 127.0.98.62:44805 requested a full tablet report, sending...
I20260924 07:10:31.372193 10622 ts_manager.cc:194] Registered new tserver with Master: 53a69ae0569e438fa37419b55212629c (127.0.98.1:33213)
I20260924 07:10:31.372866 10622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41791
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 07:10:31.402972 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44805
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:31.483238 10819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:31.483422 10819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:31.483444 10819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:31.484491 10819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:31.484524 10819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:31.484549 10819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:10:31.486014 10819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:10:31.486182 10819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:31.486344 10819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:31.491194 10827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:31.491257 10819 server_base.cc:1034] running on GCE node
W20260924 07:10:31.491220 10824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:31.491243 10825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:31.491626 10819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:31.491853 10819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:31.492998 10819 hybrid_clock.cc:648] HybridClock initialized: now 1790233831492981 us; error 42 us; skew 500 ppm
Sep 24 07:10:31 dist-test-slave-14ww krb5kdc[10578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:31.495357 10819 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:10:31.495836 10819 webserver.cc:460] Webserver started at http://127.0.98.2:36679/ using document root <none> and password file <none>
I20260924 07:10:31.496054 10819 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:31.496122 10819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:31.496269 10819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:31.497359 10819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-1/data/instance:
uuid: "9f827c9f1ae7462391b3194c3ee625d0"
format_stamp: "Formatted at 2026-09-24 07:10:31 on dist-test-slave-14ww"
server_key: "87ece8ec810be7c39df09e5389686b68"
server_key_iv: "7630c830c8190aa39c339bbf626094c5"
server_key_version: "encryptionkey@0"
I20260924 07:10:31.497715 10819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9f827c9f1ae7462391b3194c3ee625d0"
format_stamp: "Formatted at 2026-09-24 07:10:31 on dist-test-slave-14ww"
server_key: "87ece8ec810be7c39df09e5389686b68"
server_key_iv: "7630c830c8190aa39c339bbf626094c5"
server_key_version: "encryptionkey@0"
I20260924 07:10:31.499002 10819 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:10:31.499872 10834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:31.500070 10819 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:10:31.500172 10819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-1/wal
uuid: "9f827c9f1ae7462391b3194c3ee625d0"
format_stamp: "Formatted at 2026-09-24 07:10:31 on dist-test-slave-14ww"
server_key: "87ece8ec810be7c39df09e5389686b68"
server_key_iv: "7630c830c8190aa39c339bbf626094c5"
server_key_version: "encryptionkey@0"
I20260924 07:10:31.500250 10819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:31.532668 10819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:31.534590 10819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:31.534756 10819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:31.535012 10819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:10:31.535333 10819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:10:31.535388 10819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:31.535431 10819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:10:31.535460 10819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:31.541293 10819 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44835
I20260924 07:10:31.541345 10947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44835 every 8 connection(s)
I20260924 07:10:31.541690 10819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 07:10:31 dist-test-slave-14ww krb5kdc[10578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:10:31.547839 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 10819
I20260924 07:10:31.547945 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:10:31.548161 392 external_mini_cluster.cc:1428] Setting key adc6c2c6ab21cde9b7dab479a3424142
I20260924 07:10:31.551162 10662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:31.542439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37313 (local address 127.0.98.62:44805)
0924 07:10:31.542508 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:31.542510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:31.542709 (+ 199us) server_negotiation.cc:407] Connection header received
0924 07:10:31.543200 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:31.543201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:31.543212 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:31.543276 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:31.544177 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:31.544664 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:31.545616 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:31.545898 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:31.545957 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:31.548192 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:31.548194 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:31.548194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:31.548218 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:31.549649 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:31.550149 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:31.550150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:31.550150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:31.550202 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:31.550497 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:31.550498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:31.550498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:31.550644 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:31.550720 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:31.550956 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 07:10:31.551043 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 07:10:31.551167 10950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:31.542415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44805 (local address 127.0.98.2:37313)
0924 07:10:31.542563 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:31.542602 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:31.543089 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:31.543329 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:31.543332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:31.543876 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:31.544069 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:31.544073 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:31.544764 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:31.544765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:31.545539 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:31.545541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:31.545990 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:31.545991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:31.546094 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:31.546515 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:31.546522 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:31.548104 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:31.549731 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:31.549737 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:31.549739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:31.550066 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:31.550265 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:31.550266 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:31.550266 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:31.550405 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:31.550714 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:31.550718 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:31.550824 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 07:10:31.550910 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 07:10:31.551019 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
I20260924 07:10:31.551644 10948 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44805
I20260924 07:10:31.551733 10948 heartbeater.cc:464] Registering TS with master...
I20260924 07:10:31.551910 10948 heartbeater.cc:511] Master 127.0.98.62:44805 requested a full tablet report, sending...
I20260924 07:10:31.552256 10622 ts_manager.cc:194] Registered new tserver with Master: 9f827c9f1ae7462391b3194c3ee625d0 (127.0.98.2:44835)
I20260924 07:10:31.552579 10622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37313
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 07:10:31.576803 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44805
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:10:31.655943 10953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:10:31.656128 10953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:10:31.656155 10953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:10:31.657353 10953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:10:31.657395 10953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:10:31.657421 10953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:10:31.658823 10953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:10:31.659053 10953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:10:31.659278 10953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:10:31.664291 10958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:31.664347 10953 server_base.cc:1034] running on GCE node
W20260924 07:10:31.664371 10961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:10:31.664286 10959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:10:31.664647 10953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:10:31.664932 10953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:10:31.666111 10953 hybrid_clock.cc:648] HybridClock initialized: now 1790233831666089 us; error 55 us; skew 500 ppm
Sep 24 07:10:31 dist-test-slave-14ww krb5kdc[10578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:10:31.668915 10953 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:10:31.669381 10953 webserver.cc:460] Webserver started at http://127.0.98.3:45351/ using document root <none> and password file <none>
I20260924 07:10:31.669615 10953 fs_manager.cc:359] Metadata directory not provided
I20260924 07:10:31.669685 10953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:10:31.669837 10953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:10:31.670841 10953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-2/data/instance:
uuid: "c105ef4a67fd4352af132674ba5da26e"
format_stamp: "Formatted at 2026-09-24 07:10:31 on dist-test-slave-14ww"
server_key: "8f74bf90dd31da7a93bf64e4f1e66759"
server_key_iv: "503870598c997d3a49f8122051d04c7d"
server_key_version: "encryptionkey@0"
I20260924 07:10:31.671202 10953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c105ef4a67fd4352af132674ba5da26e"
format_stamp: "Formatted at 2026-09-24 07:10:31 on dist-test-slave-14ww"
server_key: "8f74bf90dd31da7a93bf64e4f1e66759"
server_key_iv: "503870598c997d3a49f8122051d04c7d"
server_key_version: "encryptionkey@0"
I20260924 07:10:31.672986 10953 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:10:31.673708 10968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:31.673930 10953 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:10:31.674036 10953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-2/wal
uuid: "c105ef4a67fd4352af132674ba5da26e"
format_stamp: "Formatted at 2026-09-24 07:10:31 on dist-test-slave-14ww"
server_key: "8f74bf90dd31da7a93bf64e4f1e66759"
server_key_iv: "503870598c997d3a49f8122051d04c7d"
server_key_version: "encryptionkey@0"
I20260924 07:10:31.674118 10953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:10:31.689648 10953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:10:31.691860 10953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:10:31.692036 10953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:10:31.692272 10953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:10:31.692572 10953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:10:31.692625 10953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:31.692662 10953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:10:31.692690 10953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:10:31.698567 10953 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33573
I20260924 07:10:31.698613 11081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33573 every 8 connection(s)
I20260924 07:10:31.698938 10953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:10:31.701210 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 10953
I20260924 07:10:31.701314 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:10:31.701499 392 external_mini_cluster.cc:1428] Setting key a55e95baf71bf050b9954ecedbcc4d73
Sep 24 07:10:31 dist-test-slave-14ww krb5kdc[10578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:10:31.709173 10662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:31.699607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43119 (local address 127.0.98.62:44805)
0924 07:10:31.699668 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:31.699670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:31.699874 (+ 204us) server_negotiation.cc:407] Connection header received
0924 07:10:31.700355 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:31.700356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:31.700367 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:31.700428 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:31.701426 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:31.702076 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:31.703152 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:31.703368 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:31.703455 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:31.705769 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:31.705771 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:31.705771 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:31.705779 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:31.707489 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:31.707947 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:31.707948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:31.707948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:31.708012 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:31.708312 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:31.708313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:31.708313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:31.708438 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:31.708546 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:31.708920 (+ 374us) server_negotiation.cc:299] Negotiation successful
0924 07:10:31.709047 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 07:10:31.709347 11084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:31.699571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44805 (local address 127.0.98.3:43119)
0924 07:10:31.699744 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:31.699785 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:31.700253 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:31.700502 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:31.700506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:31.701108 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:10:31.701295 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:31.701299 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:31.702183 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:31.702184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:31.703069 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:31.703072 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:31.703463 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:31.703464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:31.703595 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:31.704118 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:10:31.704126 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:10:31.705674 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:10:31.707607 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:31.707613 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:31.707615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:31.707839 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:31.708108 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:10:31.708109 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:10:31.708109 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:10:31.708245 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:10:31.708557 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:10:31.708563 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:10:31.708759 (+ 196us) client_negotiation.cc:769] Sending connection context
0924 07:10:31.708883 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 07:10:31.709065 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":102,"threads_started":1}
I20260924 07:10:31.710144 11082 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44805
I20260924 07:10:31.710233 11082 heartbeater.cc:464] Registering TS with master...
I20260924 07:10:31.710407 11082 heartbeater.cc:511] Master 127.0.98.62:44805 requested a full tablet report, sending...
I20260924 07:10:31.710813 10622 ts_manager.cc:194] Registered new tserver with Master: c105ef4a67fd4352af132674ba5da26e (127.0.98.3:33573)
I20260924 07:10:31.711213 10622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43119
I20260924 07:10:31.713039 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:10:31.719980 10662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:31.714235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53884 (local address 127.0.98.62:44805)
0924 07:10:31.714313 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:31.714315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:31.714365 (+ 50us) server_negotiation.cc:407] Connection header received
0924 07:10:31.714421 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:31.714421 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:31.714429 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:31.714484 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:10:31.714992 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:31.715473 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:31.716162 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:31.716334 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:31.716391 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:31.717582 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:10:31.717583 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:10:31.717584 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:10:31.717590 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:10:31.718782 (+ 1192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:31.719308 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:31.719309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:31.719309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:31.719363 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:10:31.719539 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:10:31.719540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:10:31.719540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:10:31.719664 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:10:31.719734 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:31.719816 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 07:10:31.719896 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 07:10:31.721652 10622 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53884:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:10:31.723203 10622 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:10:31.729523 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:31.725554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33213 (local address 127.0.0.1:54500)
0924 07:10:31.726065 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:31.726075 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:31.726132 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:31.726409 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:31.726410 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:31.726413 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:10:31.726460 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:31.726462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:31.727682 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:31.727683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:31.728445 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:31.728447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:31.729318 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:31.729319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:31.729374 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:31.729386 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 07:10:31.729439 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 07:10:31.729473 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":83,"threads_started":1}
I20260924 07:10:31.729571 11098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:31.725882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54500 (local address 127.0.98.1:33213)
0924 07:10:31.726052 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:31.726057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:31.726123 (+ 66us) server_negotiation.cc:407] Connection header received
0924 07:10:31.726205 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:31.726206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:31.726237 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:31.726316 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:10:31.726542 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:31.727585 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:31.728552 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:31.729239 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:31.729321 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:31.729384 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:31.729452 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 07:10:31.729489 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":91,"threads_started":1}
I20260924 07:10:31.729939 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:31.725528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33573 (local address 127.0.0.1:50954)
0924 07:10:31.725635 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:31.725673 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:31.725737 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:31.726090 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:31.726093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:31.726096 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:10:31.726177 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:31.726178 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:31.727459 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:31.727460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:31.728866 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:31.728869 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:31.729725 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:31.729726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:31.729781 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:31.729785 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:10:31.729827 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 07:10:31.729864 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":90,"thread_start_us":68,"threads_started":1}
I20260924 07:10:31.730161 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:31.725693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50954 (local address 127.0.98.3:33573)
0924 07:10:31.725904 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:31.725909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:31.725920 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:10:31.725964 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:31.725966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:31.725988 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:31.726080 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:10:31.726250 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:31.727342 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:31.728945 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:31.729645 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:31.729714 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:31.729767 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:31.730031 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 07:10:31.730080 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":84,"threads_started":1}
I20260924 07:10:31.730211 10748 tablet_service.cc:1467] Processing CreateTablet for tablet 9adf1bf70cca4906a1115901ba208cc4 (DEFAULT_TABLE table=test-table [id=f3a4a6fc46bd4ea9a7efdb5dccbac382]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:10:31.730562 10748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9adf1bf70cca4906a1115901ba208cc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:31.730793 11016 tablet_service.cc:1467] Processing CreateTablet for tablet 9adf1bf70cca4906a1115901ba208cc4 (DEFAULT_TABLE table=test-table [id=f3a4a6fc46bd4ea9a7efdb5dccbac382]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:10:31.730894 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:31.725554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44835 (local address 127.0.0.1:52196)
0924 07:10:31.726221 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:31.726229 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:31.726273 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:31.726551 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:31.726552 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:31.726553 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:10:31.726601 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:31.726602 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:31.727452 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:31.727453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:31.728867 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:31.728869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:31.730656 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:31.730657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:31.730719 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:31.730724 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:10:31.730783 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 07:10:31.730819 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":96,"thread_start_us":76,"threads_started":1}
I20260924 07:10:31.730931 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:31.725809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52196 (local address 127.0.98.2:44835)
0924 07:10:31.726001 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:31.726006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:31.726330 (+ 324us) server_negotiation.cc:407] Connection header received
0924 07:10:31.726385 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:31.726387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:31.726402 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:31.726473 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:10:31.726676 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:31.727350 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:31.729802 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:31.730493 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:31.730565 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:31.730619 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:31.730814 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 07:10:31.730859 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":91,"threads_started":1}
I20260924 07:10:31.731105 11016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9adf1bf70cca4906a1115901ba208cc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:31.733435 11099 tablet_bootstrap.cc:492] T 9adf1bf70cca4906a1115901ba208cc4 P 53a69ae0569e438fa37419b55212629c: Bootstrap starting.
I20260924 07:10:31.733454 10882 tablet_service.cc:1467] Processing CreateTablet for tablet 9adf1bf70cca4906a1115901ba208cc4 (DEFAULT_TABLE table=test-table [id=f3a4a6fc46bd4ea9a7efdb5dccbac382]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:10:31.733812 10882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9adf1bf70cca4906a1115901ba208cc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:10:31.734128 11099 tablet_bootstrap.cc:654] T 9adf1bf70cca4906a1115901ba208cc4 P 53a69ae0569e438fa37419b55212629c: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:31.734676 11099 log.cc:824] T 9adf1bf70cca4906a1115901ba208cc4 P 53a69ae0569e438fa37419b55212629c: Log is configured to *not* fsync() on all Append() calls
I20260924 07:10:31.735023 11100 tablet_bootstrap.cc:492] T 9adf1bf70cca4906a1115901ba208cc4 P c105ef4a67fd4352af132674ba5da26e: Bootstrap starting.
I20260924 07:10:31.735380 11099 tablet_bootstrap.cc:492] T 9adf1bf70cca4906a1115901ba208cc4 P 53a69ae0569e438fa37419b55212629c: No bootstrap required, opened a new log
I20260924 07:10:31.735431 11099 ts_tablet_manager.cc:1397] T 9adf1bf70cca4906a1115901ba208cc4 P 53a69ae0569e438fa37419b55212629c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:10:31.736415 11102 tablet_bootstrap.cc:492] T 9adf1bf70cca4906a1115901ba208cc4 P 9f827c9f1ae7462391b3194c3ee625d0: Bootstrap starting.
I20260924 07:10:31.737054 11102 tablet_bootstrap.cc:654] T 9adf1bf70cca4906a1115901ba208cc4 P 9f827c9f1ae7462391b3194c3ee625d0: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:31.737073 11099 raft_consensus.cc:348] T 9adf1bf70cca4906a1115901ba208cc4 P 53a69ae0569e438fa37419b55212629c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c105ef4a67fd4352af132674ba5da26e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33573 } } peers { permanent_uuid: "53a69ae0569e438fa37419b55212629c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33213 } } peers { permanent_uuid: "9f827c9f1ae7462391b3194c3ee625d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44835 } }
I20260924 07:10:31.737255 11099 raft_consensus.cc:374] T 9adf1bf70cca4906a1115901ba208cc4 P 53a69ae0569e438fa37419b55212629c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:31.737267 11100 tablet_bootstrap.cc:654] T 9adf1bf70cca4906a1115901ba208cc4 P c105ef4a67fd4352af132674ba5da26e: Neither blocks nor log segments found. Creating new log.
I20260924 07:10:31.737308 11099 raft_consensus.cc:729] T 9adf1bf70cca4906a1115901ba208cc4 P 53a69ae0569e438fa37419b55212629c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53a69ae0569e438fa37419b55212629c, State: Initialized, Role: FOLLOWER
I20260924 07:10:31.737448 11099 consensus_queue.cc:260] T 9adf1bf70cca4906a1115901ba208cc4 P 53a69ae0569e438fa37419b55212629c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c105ef4a67fd4352af132674ba5da26e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33573 } } peers { permanent_uuid: "53a69ae0569e438fa37419b55212629c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33213 } } peers { permanent_uuid: "9f827c9f1ae7462391b3194c3ee625d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44835 } }
I20260924 07:10:31.737707 11099 ts_tablet_manager.cc:1428] T 9adf1bf70cca4906a1115901ba208cc4 P 53a69ae0569e438fa37419b55212629c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 07:10:31.737721 11102 log.cc:824] T 9adf1bf70cca4906a1115901ba208cc4 P 9f827c9f1ae7462391b3194c3ee625d0: Log is configured to *not* fsync() on all Append() calls
I20260924 07:10:31.737823 10814 heartbeater.cc:503] Master 127.0.98.62:44805 was elected leader, sending a full tablet report...
I20260924 07:10:31.737887 11100 log.cc:824] T 9adf1bf70cca4906a1115901ba208cc4 P c105ef4a67fd4352af132674ba5da26e: Log is configured to *not* fsync() on all Append() calls
I20260924 07:10:31.738535 11102 tablet_bootstrap.cc:492] T 9adf1bf70cca4906a1115901ba208cc4 P 9f827c9f1ae7462391b3194c3ee625d0: No bootstrap required, opened a new log
I20260924 07:10:31.738605 11102 ts_tablet_manager.cc:1397] T 9adf1bf70cca4906a1115901ba208cc4 P 9f827c9f1ae7462391b3194c3ee625d0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:10:31.738842 11100 tablet_bootstrap.cc:492] T 9adf1bf70cca4906a1115901ba208cc4 P c105ef4a67fd4352af132674ba5da26e: No bootstrap required, opened a new log
I20260924 07:10:31.738919 11100 ts_tablet_manager.cc:1397] T 9adf1bf70cca4906a1115901ba208cc4 P c105ef4a67fd4352af132674ba5da26e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 07:10:31.740079 11102 raft_consensus.cc:348] T 9adf1bf70cca4906a1115901ba208cc4 P 9f827c9f1ae7462391b3194c3ee625d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c105ef4a67fd4352af132674ba5da26e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33573 } } peers { permanent_uuid: "53a69ae0569e438fa37419b55212629c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33213 } } peers { permanent_uuid: "9f827c9f1ae7462391b3194c3ee625d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44835 } }
I20260924 07:10:31.740202 11102 raft_consensus.cc:374] T 9adf1bf70cca4906a1115901ba208cc4 P 9f827c9f1ae7462391b3194c3ee625d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:31.740223 11102 raft_consensus.cc:729] T 9adf1bf70cca4906a1115901ba208cc4 P 9f827c9f1ae7462391b3194c3ee625d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f827c9f1ae7462391b3194c3ee625d0, State: Initialized, Role: FOLLOWER
I20260924 07:10:31.740337 11102 consensus_queue.cc:260] T 9adf1bf70cca4906a1115901ba208cc4 P 9f827c9f1ae7462391b3194c3ee625d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c105ef4a67fd4352af132674ba5da26e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33573 } } peers { permanent_uuid: "53a69ae0569e438fa37419b55212629c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33213 } } peers { permanent_uuid: "9f827c9f1ae7462391b3194c3ee625d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44835 } }
I20260924 07:10:31.740381 11100 raft_consensus.cc:348] T 9adf1bf70cca4906a1115901ba208cc4 P c105ef4a67fd4352af132674ba5da26e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c105ef4a67fd4352af132674ba5da26e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33573 } } peers { permanent_uuid: "53a69ae0569e438fa37419b55212629c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33213 } } peers { permanent_uuid: "9f827c9f1ae7462391b3194c3ee625d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44835 } }
I20260924 07:10:31.740561 11100 raft_consensus.cc:374] T 9adf1bf70cca4906a1115901ba208cc4 P c105ef4a67fd4352af132674ba5da26e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:10:31.740603 11100 raft_consensus.cc:729] T 9adf1bf70cca4906a1115901ba208cc4 P c105ef4a67fd4352af132674ba5da26e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c105ef4a67fd4352af132674ba5da26e, State: Initialized, Role: FOLLOWER
I20260924 07:10:31.740612 11102 ts_tablet_manager.cc:1428] T 9adf1bf70cca4906a1115901ba208cc4 P 9f827c9f1ae7462391b3194c3ee625d0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:10:31.740621 10948 heartbeater.cc:503] Master 127.0.98.62:44805 was elected leader, sending a full tablet report...
I20260924 07:10:31.740717 11100 consensus_queue.cc:260] T 9adf1bf70cca4906a1115901ba208cc4 P c105ef4a67fd4352af132674ba5da26e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c105ef4a67fd4352af132674ba5da26e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33573 } } peers { permanent_uuid: "53a69ae0569e438fa37419b55212629c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33213 } } peers { permanent_uuid: "9f827c9f1ae7462391b3194c3ee625d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44835 } }
I20260924 07:10:31.740998 11100 ts_tablet_manager.cc:1428] T 9adf1bf70cca4906a1115901ba208cc4 P c105ef4a67fd4352af132674ba5da26e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 07:10:31.741082 11082 heartbeater.cc:503] Master 127.0.98.62:44805 was elected leader, sending a full tablet report...
W20260924 07:10:31.792263 10949 tablet.cc:2369] T 9adf1bf70cca4906a1115901ba208cc4 P 9f827c9f1ae7462391b3194c3ee625d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:10:31.813534 11103 raft_consensus.cc:482] T 9adf1bf70cca4906a1115901ba208cc4 P 53a69ae0569e438fa37419b55212629c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:10:31.813654 11103 raft_consensus.cc:504] T 9adf1bf70cca4906a1115901ba208cc4 P 53a69ae0569e438fa37419b55212629c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c105ef4a67fd4352af132674ba5da26e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33573 } } peers { permanent_uuid: "53a69ae0569e438fa37419b55212629c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33213 } } peers { permanent_uuid: "9f827c9f1ae7462391b3194c3ee625d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44835 } }
I20260924 07:10:31.813983 11103 leader_election.cc:288] T 9adf1bf70cca4906a1115901ba208cc4 P 53a69ae0569e438fa37419b55212629c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c105ef4a67fd4352af132674ba5da26e (127.0.98.3:33573), 9f827c9f1ae7462391b3194c3ee625d0 (127.0.98.2:44835)
I20260924 07:10:31.816840 10816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:31.814191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33573 (local address 127.0.98.1:49501)
0924 07:10:31.814245 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:31.814259 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:31.814313 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:31.814495 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:31.814496 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:31.814498 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:10:31.814568 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:31.814570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:31.815234 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:31.815235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:31.815982 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:31.815984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:31.816544 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:31.816545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:31.816623 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:31.816628 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:10:31.816702 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 07:10:31.816762 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 07:10:31.816833 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:31.814328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49501 (local address 127.0.98.3:33573)
0924 07:10:31.814372 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:31.814374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:31.814381 (+ 7us) server_negotiation.cc:407] Connection header received
0924 07:10:31.814413 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:31.814413 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:31.814422 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:31.814489 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:10:31.814651 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:31.815147 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:31.816078 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:31.816457 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:31.816544 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:31.816554 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:31.816708 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 07:10:31.816770 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 07:10:31.816924 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:31.814259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41279 (local address 127.0.98.2:44835)
0924 07:10:31.814310 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:31.814312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:31.814461 (+ 149us) server_negotiation.cc:407] Connection header received
0924 07:10:31.814536 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:31.814537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:31.814549 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:31.814622 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:10:31.814784 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:31.815244 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:31.816166 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:31.816543 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:31.816627 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:31.816637 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:31.816823 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 07:10:31.816866 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 07:10:31.816933 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:31.814191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44835 (local address 127.0.98.1:41279)
0924 07:10:31.814375 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:31.814394 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 07:10:31.814470 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:10:31.814638 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:10:31.814638 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:10:31.814640 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:10:31.814714 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:31.814715 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:31.815331 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:31.815332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:31.816068 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:10:31.816070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:10:31.816627 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:10:31.816628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:10:31.816706 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:31.816709 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:10:31.816814 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 07:10:31.816852 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":10,"thread_start_us":124,"threads_started":1}
I20260924 07:10:31.817168 11036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9adf1bf70cca4906a1115901ba208cc4" candidate_uuid: "53a69ae0569e438fa37419b55212629c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c105ef4a67fd4352af132674ba5da26e" is_pre_election: true
I20260924 07:10:31.817250 10902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9adf1bf70cca4906a1115901ba208cc4" candidate_uuid: "53a69ae0569e438fa37419b55212629c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f827c9f1ae7462391b3194c3ee625d0" is_pre_election: true
I20260924 07:10:31.817322 11036 raft_consensus.cc:2454] T 9adf1bf70cca4906a1115901ba208cc4 P c105ef4a67fd4352af132674ba5da26e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53a69ae0569e438fa37419b55212629c in term 0.
I20260924 07:10:31.817399 10902 raft_consensus.cc:2454] T 9adf1bf70cca4906a1115901ba208cc4 P 9f827c9f1ae7462391b3194c3ee625d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53a69ae0569e438fa37419b55212629c in term 0.
I20260924 07:10:31.817546 10703 leader_election.cc:302] T 9adf1bf70cca4906a1115901ba208cc4 P 53a69ae0569e438fa37419b55212629c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53a69ae0569e438fa37419b55212629c, c105ef4a67fd4352af132674ba5da26e; no voters:
I20260924 07:10:31.817673 11103 raft_consensus.cc:2781] T 9adf1bf70cca4906a1115901ba208cc4 P 53a69ae0569e438fa37419b55212629c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:10:31.817718 11103 raft_consensus.cc:482] T 9adf1bf70cca4906a1115901ba208cc4 P 53a69ae0569e438fa37419b55212629c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:10:31.817739 11103 raft_consensus.cc:3037] T 9adf1bf70cca4906a1115901ba208cc4 P 53a69ae0569e438fa37419b55212629c [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:31.818497 11103 raft_consensus.cc:504] T 9adf1bf70cca4906a1115901ba208cc4 P 53a69ae0569e438fa37419b55212629c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c105ef4a67fd4352af132674ba5da26e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33573 } } peers { permanent_uuid: "53a69ae0569e438fa37419b55212629c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33213 } } peers { permanent_uuid: "9f827c9f1ae7462391b3194c3ee625d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44835 } }
I20260924 07:10:31.818635 11103 leader_election.cc:288] T 9adf1bf70cca4906a1115901ba208cc4 P 53a69ae0569e438fa37419b55212629c [CANDIDATE]: Term 1 election: Requested vote from peers c105ef4a67fd4352af132674ba5da26e (127.0.98.3:33573), 9f827c9f1ae7462391b3194c3ee625d0 (127.0.98.2:44835)
I20260924 07:10:31.818820 11036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9adf1bf70cca4906a1115901ba208cc4" candidate_uuid: "53a69ae0569e438fa37419b55212629c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c105ef4a67fd4352af132674ba5da26e"
I20260924 07:10:31.818783 10902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9adf1bf70cca4906a1115901ba208cc4" candidate_uuid: "53a69ae0569e438fa37419b55212629c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f827c9f1ae7462391b3194c3ee625d0"
I20260924 07:10:31.818910 11036 raft_consensus.cc:3037] T 9adf1bf70cca4906a1115901ba208cc4 P c105ef4a67fd4352af132674ba5da26e [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:31.818907 10902 raft_consensus.cc:3037] T 9adf1bf70cca4906a1115901ba208cc4 P 9f827c9f1ae7462391b3194c3ee625d0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:10:31.819737 11036 raft_consensus.cc:2454] T 9adf1bf70cca4906a1115901ba208cc4 P c105ef4a67fd4352af132674ba5da26e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53a69ae0569e438fa37419b55212629c in term 1.
I20260924 07:10:31.819876 10902 raft_consensus.cc:2454] T 9adf1bf70cca4906a1115901ba208cc4 P 9f827c9f1ae7462391b3194c3ee625d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53a69ae0569e438fa37419b55212629c in term 1.
I20260924 07:10:31.819941 10703 leader_election.cc:302] T 9adf1bf70cca4906a1115901ba208cc4 P 53a69ae0569e438fa37419b55212629c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53a69ae0569e438fa37419b55212629c, c105ef4a67fd4352af132674ba5da26e; no voters:
I20260924 07:10:31.820024 11103 raft_consensus.cc:2781] T 9adf1bf70cca4906a1115901ba208cc4 P 53a69ae0569e438fa37419b55212629c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:10:31.820176 11103 raft_consensus.cc:686] T 9adf1bf70cca4906a1115901ba208cc4 P 53a69ae0569e438fa37419b55212629c [term 1 LEADER]: Becoming Leader. State: Replica: 53a69ae0569e438fa37419b55212629c, State: Running, Role: LEADER
I20260924 07:10:31.820278 11103 consensus_queue.cc:237] T 9adf1bf70cca4906a1115901ba208cc4 P 53a69ae0569e438fa37419b55212629c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c105ef4a67fd4352af132674ba5da26e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33573 } } peers { permanent_uuid: "53a69ae0569e438fa37419b55212629c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33213 } } peers { permanent_uuid: "9f827c9f1ae7462391b3194c3ee625d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44835 } }
I20260924 07:10:31.821131 10623 catalog_manager.cc:5495] T 9adf1bf70cca4906a1115901ba208cc4 P 53a69ae0569e438fa37419b55212629c reported cstate change: term changed from 0 to 1, leader changed from <none> to 53a69ae0569e438fa37419b55212629c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "53a69ae0569e438fa37419b55212629c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c105ef4a67fd4352af132674ba5da26e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53a69ae0569e438fa37419b55212629c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33213 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f827c9f1ae7462391b3194c3ee625d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44835 } health_report { overall_health: UNKNOWN } } }
I20260924 07:10:31.845396 11098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:31.843031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54514 (local address 127.0.98.1:33213)
0924 07:10:31.843073 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 07:10:31.843075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:10:31.843094 (+ 19us) server_negotiation.cc:407] Connection header received
0924 07:10:31.843124 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:10:31.843124 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:10:31.843141 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:10:31.843201 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:10:31.843294 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:31.843742 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:31.844506 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:10:31.844690 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:10:31.844782 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:10:31.845003 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:10:31.845095 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:10:31.845222 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 07:10:31.845262 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 07:10:31.845279 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 07:10:31.847111 10902 raft_consensus.cc:1261] T 9adf1bf70cca4906a1115901ba208cc4 P 9f827c9f1ae7462391b3194c3ee625d0 [term 1 FOLLOWER]: Refusing update from remote peer 53a69ae0569e438fa37419b55212629c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:10:31.847188 11036 raft_consensus.cc:1261] T 9adf1bf70cca4906a1115901ba208cc4 P c105ef4a67fd4352af132674ba5da26e [term 1 FOLLOWER]: Refusing update from remote peer 53a69ae0569e438fa37419b55212629c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:10:31.847450 11109 consensus_queue.cc:1035] T 9adf1bf70cca4906a1115901ba208cc4 P 53a69ae0569e438fa37419b55212629c [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f827c9f1ae7462391b3194c3ee625d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:10:31.847584 11103 consensus_queue.cc:1035] T 9adf1bf70cca4906a1115901ba208cc4 P 53a69ae0569e438fa37419b55212629c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c105ef4a67fd4352af132674ba5da26e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:10:31.851956 10623 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:10:31.852047 10623 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:10:31.852756 10623 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ba5bb29d44e54b0a8ec12298899dbe34: Sending DeleteTablet for 3 replicas of tablet 9adf1bf70cca4906a1115901ba208cc4
I20260924 07:10:31.853045 11016 tablet_service.cc:1514] Processing DeleteTablet for tablet 9adf1bf70cca4906a1115901ba208cc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:10:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50954
I20260924 07:10:31.853091 10882 tablet_service.cc:1514] Processing DeleteTablet for tablet 9adf1bf70cca4906a1115901ba208cc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:10:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52196
I20260924 07:10:31.853157 10748 tablet_service.cc:1514] Processing DeleteTablet for tablet 9adf1bf70cca4906a1115901ba208cc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:10:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54500
I20260924 07:10:31.853353 11121 tablet_replica.cc:330] T 9adf1bf70cca4906a1115901ba208cc4 P c105ef4a67fd4352af132674ba5da26e: stopping tablet replica
I20260924 07:10:31.853350 11123 tablet_replica.cc:330] T 9adf1bf70cca4906a1115901ba208cc4 P 53a69ae0569e438fa37419b55212629c: stopping tablet replica
I20260924 07:10:31.853451 11121 raft_consensus.cc:2229] T 9adf1bf70cca4906a1115901ba208cc4 P c105ef4a67fd4352af132674ba5da26e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:10:31.853535 11123 raft_consensus.cc:2229] T 9adf1bf70cca4906a1115901ba208cc4 P 53a69ae0569e438fa37419b55212629c [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:10:31.853536 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 10685
I20260924 07:10:31.853552 11121 raft_consensus.cc:2258] T 9adf1bf70cca4906a1115901ba208cc4 P c105ef4a67fd4352af132674ba5da26e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:10:31.855105 11122 tablet_replica.cc:330] T 9adf1bf70cca4906a1115901ba208cc4 P 9f827c9f1ae7462391b3194c3ee625d0: stopping tablet replica
I20260924 07:10:31.855225 11122 raft_consensus.cc:2229] T 9adf1bf70cca4906a1115901ba208cc4 P 9f827c9f1ae7462391b3194c3ee625d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:10:31.855290 11122 raft_consensus.cc:2258] T 9adf1bf70cca4906a1115901ba208cc4 P 9f827c9f1ae7462391b3194c3ee625d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:10:31.855629 11122 ts_tablet_manager.cc:1905] T 9adf1bf70cca4906a1115901ba208cc4 P 9f827c9f1ae7462391b3194c3ee625d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:10:31.856051 11121 ts_tablet_manager.cc:1905] T 9adf1bf70cca4906a1115901ba208cc4 P c105ef4a67fd4352af132674ba5da26e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:10:31.857472 11122 ts_tablet_manager.cc:1918] T 9adf1bf70cca4906a1115901ba208cc4 P 9f827c9f1ae7462391b3194c3ee625d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:10:31.857548 11122 log.cc:1196] T 9adf1bf70cca4906a1115901ba208cc4 P 9f827c9f1ae7462391b3194c3ee625d0: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-1/wal/wals/9adf1bf70cca4906a1115901ba208cc4
I20260924 07:10:31.857695 11121 ts_tablet_manager.cc:1918] T 9adf1bf70cca4906a1115901ba208cc4 P c105ef4a67fd4352af132674ba5da26e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:10:31.857764 11121 log.cc:1196] T 9adf1bf70cca4906a1115901ba208cc4 P c105ef4a67fd4352af132674ba5da26e: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790233772407728-392-0/minicluster-data/ts-2/wal/wals/9adf1bf70cca4906a1115901ba208cc4
I20260924 07:10:31.857833 11122 ts_tablet_manager.cc:1939] T 9adf1bf70cca4906a1115901ba208cc4 P 9f827c9f1ae7462391b3194c3ee625d0: Deleting consensus metadata
I20260924 07:10:31.858033 11121 ts_tablet_manager.cc:1939] T 9adf1bf70cca4906a1115901ba208cc4 P c105ef4a67fd4352af132674ba5da26e: Deleting consensus metadata
I20260924 07:10:31.858211 10607 catalog_manager.cc:4841] TS 9f827c9f1ae7462391b3194c3ee625d0 (127.0.98.2:44835): tablet 9adf1bf70cca4906a1115901ba208cc4 (table test-table [id=f3a4a6fc46bd4ea9a7efdb5dccbac382]) successfully deleted
I20260924 07:10:31.858386 10609 catalog_manager.cc:4841] TS c105ef4a67fd4352af132674ba5da26e (127.0.98.3:33573): tablet 9adf1bf70cca4906a1115901ba208cc4 (table test-table [id=f3a4a6fc46bd4ea9a7efdb5dccbac382]) successfully deleted
W20260924 07:10:31.859411 10608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33213 (error 108)
I20260924 07:10:31.859640 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 10819
I20260924 07:10:31.864353 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:10:31.864207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33213 (local address 127.0.0.1:54518)
0924 07:10:31.864279 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0924 07:10:31.864313 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 07:10:31.864341 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 10953
W20260924 07:10:31.868678 10608 catalog_manager.cc:4584] TS 53a69ae0569e438fa37419b55212629c (127.0.98.1:33213): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9adf1bf70cca4906a1115901ba208cc4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33213: connect: Connection refused (error 111)
I20260924 07:10:31.868796 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 10591
2026-09-24T07:10:31Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2017 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:10:31 dist-test-slave-14ww krb5kdc[11127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:10:31 dist-test-slave-14ww krb5kdc[11127](info): set up 2 sockets
Sep 24 07:10:31 dist-test-slave-14ww krb5kdc[11127](info): commencing operation
krb5kdc: starting...
W20260924 07:10:32.907971 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:10:33 dist-test-slave-14ww krb5kdc[11127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:10:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:10:33Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 07:10:33.085677 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskIclK1b/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/postgres -l logfile start
2026-09-24 07:10:35.650 UTC [11157] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 07:10:35.650 UTC [11157] LOG: listening on IPv4 address "127.0.98.20", port 49115
2026-09-24 07:10:35.654 UTC [11157] LOG: listening on Unix socket "/tmp/.s.PGSQL.49115"
2026-09-24 07:10:35.676 UTC [11160] LOG: database system was shut down at 2026-09-24 07:10:34 UTC
2026-09-24 07:10:35.680 UTC [11157] LOG: database system is ready to accept connections
I20260924 07:10:36.625916 392 mini_postgres.cc:96] Postgres bound to 49115
2026-09-24 07:10:36.631 UTC [11168] FATAL: database "slave" does not exist
127.0.98.20:49115 - accepting connections
I20260924 07:10:36.632305 392 mini_ranger.cc:161] Starting Ranger...
I20260924 07:10:36.646486 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 07:10:37.141963 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 07:10:37.142056 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ranger-admin
2026-09-24 07:10:37,431 [I] DB FLAVOR :POSTGRES
2026-09-24 07:10:37,431 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 07:10:37,431 [I] Checking connection..
2026-09-24 07:10:37,431 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 07:10:37,674 [I] Checking connection passed.
2026-09-24 07:10:37,675 [I] --------- Verifying version history table ---------
2026-09-24 07:10:37,675 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 07:10:37,919 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 07:10:37,919 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 07:10:38,150 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 07:10:38,150 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 07:10:38,150 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49115/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 07:10:38.386 UTC [11232] WARNING: there is no transaction in progress
2026-09-24 07:10:38,392 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 07:10:38,393 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 07:10:38,618 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 07:10:38,618 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 07:10:38,619 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 07:10:38,841 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 07:10:39,063 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-14ww.c.gcp-upstream.internal','N') ;"
2026-09-24 07:10:39,275 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 07:10:39,517 [I] Table x_portal_user does not exist in database ranger
2026-09-24 07:10:39,518 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 07:10:39,771 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 07:10:39,771 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 07:10:40,020 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 07:10:40,263 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 07:10:40,264 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49115/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 07:10:41.227 UTC [11340] WARNING: there is no transaction in progress
2026-09-24 07:10:41.243 UTC [11340] WARNING: there is no transaction in progress
2026-09-24 07:10:41.258 UTC [11340] WARNING: there is no transaction in progress
2026-09-24 07:10:41.273 UTC [11340] WARNING: there is no transaction in progress
2026-09-24 07:10:41.289 UTC [11340] WARNING: there is no transaction in progress
2026-09-24 07:10:41.305 UTC [11340] WARNING: there is no transaction in progress
2026-09-24 07:10:41.359 UTC [11340] WARNING: there is no transaction in progress
2026-09-24 07:10:41.366 UTC [11340] WARNING: there is no transaction in progress
2026-09-24 07:10:41.380 UTC [11340] WARNING: there is no transaction in progress
2026-09-24 07:10:41.390 UTC [11340] WARNING: there is no transaction in progress
2026-09-24 07:10:41.401 UTC [11340] WARNING: there is no transaction in progress
2026-09-24 07:10:41.415 UTC [11340] WARNING: there is no transaction in progress
2026-09-24 07:10:41.424 UTC [11340] WARNING: there is no transaction in progress
2026-09-24 07:10:41.431 UTC [11340] WARNING: there is no transaction in progress
2026-09-24 07:10:41.438 UTC [11340] WARNING: there is no transaction in progress
2026-09-24 07:10:41.445 UTC [11340] WARNING: there is no transaction in progress
2026-09-24 07:10:42,131 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 07:10:42,131 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 07:10:42,385 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 07:10:42,386 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 07:10:42,636 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 07:10:42,636 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 07:10:42,859 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 07:10:42,860 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 07:10:43,122 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 07:10:43,374 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-14ww.c.gcp-upstream.internal';"
2026-09-24 07:10:43,610 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 07:10:43,610 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49115/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 07:10:43,845 [I] DB_PATCHES have already been applied
I20260924 07:10:43.850713 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskIclK1b/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 07:10:43.850795 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 07:10:43.853768 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:41427
Sep 24, 2026 7:10:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:10:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:10:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 7:10:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 7:10:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 7:10:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 7:10:44 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 07:10:44.952653 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.320s user 0.001s sys 0.008s
Sep 24, 2026 7:11:03 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 7:11:03 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 7:11:03 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 7:11:03 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 7:11:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 7:11:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 7:11:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 7:11:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 7:11:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 7:11:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 7:11:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 7:11:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 7:11:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 7:11:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 7:11:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 7:11:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 7:11:03 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 7:11:04 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 07:11:05.153261 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 07:11:05 dist-test-slave-14ww krb5kdc[11127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233865, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 07:11:05.235476 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 07:11:05.244992 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 07:11:06.136992 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 07:11:06.137079 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ranger-kms
I20260924 07:11:06.137100 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:49115
2026-09-24 07:11:06,309 [I] DB FLAVOR :POSTGRES
2026-09-24 07:11:06,310 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 07:11:06,310 [I] Checking connection
2026-09-24 07:11:06,310 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49115/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 07:11:06,565 [I] connection success
2026-09-24 07:11:06,565 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 07:11:06,565 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 07:11:06,566 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49115/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 07:11:06,809 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 07:11:06,809 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 07:11:06,810 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 07:11:06,810 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:49115/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 07:11:07,059 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 07:11:07.505327 392 mini_ranger_kms.cc:310] Created kms service
I20260924 07:11:07.679173 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 07:11:07.720558 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 07:11:07 dist-test-slave-14ww krb5kdc[11127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233865, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 07:11:08.121628 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 07:11:08.121905 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 07:11:08.121937 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 07:11:08.125427 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11520
I20260924 07:11:08.125530 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:57491
Sep 24, 2026 7:11:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:11:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:11:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 7:11:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 7:11:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 7:11:09 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 07:11:10.399159 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.164s user 0.017s sys 0.137s
I20260924 07:11:10.399286 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 07:11:10.399398 392 mini_ranger_kms.cc:401] 127.0.98.20:57491/kms/v1/keys
Sep 24 07:11:10 dist-test-slave-14ww krb5kdc[11127](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790233870, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 07:11:11 dist-test-slave-14ww krb5kdc[11127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233870, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 07:11:11 dist-test-slave-14ww krb5kdc[11127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233870, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 07:11:13 dist-test-slave-14ww krb5kdc[11127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:11:13.772053 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44583
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44583
--ranger_config_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:57491/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:13.853389 11571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:13.853574 11571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:13.853595 11571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:13.854699 11571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:11:13.854738 11571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:13.854753 11571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:13.854764 11571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:11:13.854776 11571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:11:13.856293 11571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:57491/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44583
--ranger_config_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:11:13.856550 11571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:13.856797 11571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:13.861824 11576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:13.861824 11579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:13.861930 11571 server_base.cc:1034] running on GCE node
W20260924 07:11:13.862267 11577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:13.862555 11571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:13.862784 11571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:13.863920 11571 hybrid_clock.cc:648] HybridClock initialized: now 1790233873863909 us; error 28 us; skew 500 ppm
Sep 24 07:11:13 dist-test-slave-14ww krb5kdc[11127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:13.866328 11571 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:11:13.866744 11571 webserver.cc:460] Webserver started at http://127.0.98.62:46625/ using document root <none> and password file <none>
I20260924 07:11:13.866947 11571 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:13.866998 11571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:13.867081 11571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 07:11:13 dist-test-slave-14ww krb5kdc[11127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 07:11:13.923039 11571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "498fcfb79c3c4a42a505f58654d88800"
format_stamp: "Formatted at 2026-09-24 07:11:13 on dist-test-slave-14ww"
server_key: "4c7cb7b25d264bfd2878e579b6b76f60"
server_key_iv: "43ff6c8b374a69971e806ffd43c8d6e4"
server_key_version: "kuduclusterkey@0"
I20260924 07:11:13.923511 11571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "498fcfb79c3c4a42a505f58654d88800"
format_stamp: "Formatted at 2026-09-24 07:11:13 on dist-test-slave-14ww"
server_key: "4c7cb7b25d264bfd2878e579b6b76f60"
server_key_iv: "43ff6c8b374a69971e806ffd43c8d6e4"
server_key_version: "kuduclusterkey@0"
I20260924 07:11:13.925318 11571 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260924 07:11:13.936764 11586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:13.936964 11571 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 07:11:13.937063 11571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "498fcfb79c3c4a42a505f58654d88800"
format_stamp: "Formatted at 2026-09-24 07:11:13 on dist-test-slave-14ww"
server_key: "4c7cb7b25d264bfd2878e579b6b76f60"
server_key_iv: "43ff6c8b374a69971e806ffd43c8d6e4"
server_key_version: "kuduclusterkey@0"
I20260924 07:11:13.937178 11571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:13.947821 11571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:13.950266 11571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:13.950430 11571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:13.955706 11571 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44583
I20260924 07:11:13.956328 11571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:11:13.959841 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 11571
I20260924 07:11:13.959942 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:11:13.960158 392 external_mini_cluster.cc:1428] Setting key 66569d98770c61d70252cf539c9d454a
I20260924 07:11:13.964892 11638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44583 every 8 connection(s)
I20260924 07:11:13.965685 11639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:13.968902 11639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 498fcfb79c3c4a42a505f58654d88800: Bootstrap starting.
I20260924 07:11:13.969511 11639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 498fcfb79c3c4a42a505f58654d88800: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:13.969825 11639 log.cc:824] T 00000000000000000000000000000000 P 498fcfb79c3c4a42a505f58654d88800: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:13.970603 11639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 498fcfb79c3c4a42a505f58654d88800: No bootstrap required, opened a new log
Sep 24 07:11:13 dist-test-slave-14ww krb5kdc[11127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:13.972697 11639 raft_consensus.cc:348] T 00000000000000000000000000000000 P 498fcfb79c3c4a42a505f58654d88800 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "498fcfb79c3c4a42a505f58654d88800" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44583 } }
I20260924 07:11:13.972890 11639 raft_consensus.cc:374] T 00000000000000000000000000000000 P 498fcfb79c3c4a42a505f58654d88800 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:13.972968 11639 raft_consensus.cc:729] T 00000000000000000000000000000000 P 498fcfb79c3c4a42a505f58654d88800 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 498fcfb79c3c4a42a505f58654d88800, State: Initialized, Role: FOLLOWER
I20260924 07:11:13.973121 11639 consensus_queue.cc:260] T 00000000000000000000000000000000 P 498fcfb79c3c4a42a505f58654d88800 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "498fcfb79c3c4a42a505f58654d88800" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44583 } }
I20260924 07:11:13.973244 11639 raft_consensus.cc:388] T 00000000000000000000000000000000 P 498fcfb79c3c4a42a505f58654d88800 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:11:13.973306 11639 raft_consensus.cc:482] T 00000000000000000000000000000000 P 498fcfb79c3c4a42a505f58654d88800 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:11:13.973358 11639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 498fcfb79c3c4a42a505f58654d88800 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:13.974007 11639 raft_consensus.cc:504] T 00000000000000000000000000000000 P 498fcfb79c3c4a42a505f58654d88800 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "498fcfb79c3c4a42a505f58654d88800" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44583 } }
I20260924 07:11:13.974128 11639 leader_election.cc:302] T 00000000000000000000000000000000 P 498fcfb79c3c4a42a505f58654d88800 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 498fcfb79c3c4a42a505f58654d88800; no voters:
I20260924 07:11:13.974291 11639 leader_election.cc:288] T 00000000000000000000000000000000 P 498fcfb79c3c4a42a505f58654d88800 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:11:13.974332 11644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 498fcfb79c3c4a42a505f58654d88800 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:11:13.974462 11644 raft_consensus.cc:686] T 00000000000000000000000000000000 P 498fcfb79c3c4a42a505f58654d88800 [term 1 LEADER]: Becoming Leader. State: Replica: 498fcfb79c3c4a42a505f58654d88800, State: Running, Role: LEADER
I20260924 07:11:13.974550 11644 consensus_queue.cc:237] T 00000000000000000000000000000000 P 498fcfb79c3c4a42a505f58654d88800 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "498fcfb79c3c4a42a505f58654d88800" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44583 } }
I20260924 07:11:13.974556 11639 sys_catalog.cc:564] T 00000000000000000000000000000000 P 498fcfb79c3c4a42a505f58654d88800 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:11:13.974886 11645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 498fcfb79c3c4a42a505f58654d88800 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "498fcfb79c3c4a42a505f58654d88800" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "498fcfb79c3c4a42a505f58654d88800" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44583 } } }
I20260924 07:11:13.974978 11645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 498fcfb79c3c4a42a505f58654d88800 [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:13.974877 11644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 498fcfb79c3c4a42a505f58654d88800 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 498fcfb79c3c4a42a505f58654d88800. Latest consensus state: current_term: 1 leader_uuid: "498fcfb79c3c4a42a505f58654d88800" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "498fcfb79c3c4a42a505f58654d88800" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44583 } } }
I20260924 07:11:13.975164 11644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 498fcfb79c3c4a42a505f58654d88800 [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:13.975844 11650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:11:13.976037 11639 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 07:11:13.976449 11650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:11:13.977979 11650 catalog_manager.cc:1347] Generated new cluster ID: 332d0a6aafa94ea5a9c85b9fbc3453ec
I20260924 07:11:13.978039 11650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:11:13.983068 11642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:13.965160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50932 (local address 127.0.98.62:44583)
0924 07:11:13.965341 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:13.965350 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:13.965378 (+ 28us) server_negotiation.cc:407] Connection header received
0924 07:11:13.965932 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:13.965941 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:13.966015 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:13.966570 (+ 555us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:13.967334 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:13.968693 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:13.969786 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:13.970304 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:13.970413 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:13.973224 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:13.973228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:13.973228 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:13.973240 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:13.976190 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:13.976592 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:13.976595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:13.976595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:13.976687 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:13.976905 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:13.976906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:13.976906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:13.977212 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:13.977334 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:13.977797 (+ 463us) server_negotiation.cc:299] Negotiation successful
0924 07:11:13.977909 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":91,"threads_started":1}
I20260924 07:11:13.997030 11650 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:11:13.997728 11650 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:11:14.005299 11650 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 498fcfb79c3c4a42a505f58654d88800: Generated new TSK 0
I20260924 07:11:14.005461 11650 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 07:11:14.640164 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:57491/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44583
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:14.738445 11684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:14.738705 11684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:14.738739 11684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:14.740342 11684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:14.740394 11684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:14.740432 11684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:11:14.742439 11684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:57491/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:11:14.742650 11684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:14.742911 11684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:14.748952 11689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:14.748960 11692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:14.749193 11684 server_base.cc:1034] running on GCE node
W20260924 07:11:14.749234 11690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:14.751911 11684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:14.752148 11684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:14.753300 11684 hybrid_clock.cc:648] HybridClock initialized: now 1790233874753289 us; error 32 us; skew 500 ppm
Sep 24 07:11:14 dist-test-slave-14ww krb5kdc[11127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:14.755741 11684 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:11:14.756178 11684 webserver.cc:460] Webserver started at http://127.0.98.1:36745/ using document root <none> and password file <none>
I20260924 07:11:14.756392 11684 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:14.756457 11684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:14.756596 11684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 07:11:14 dist-test-slave-14ww krb5kdc[11127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 07:11:14.769757 11684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-0/data/instance:
uuid: "074aef5591034ef8ad8f6ebf7648f8f3"
format_stamp: "Formatted at 2026-09-24 07:11:14 on dist-test-slave-14ww"
server_key: "34aab3afe45c4e510e6ddf80a6b97097"
server_key_iv: "680a0fc92094b1fae4e0f4035bb79d9f"
server_key_version: "kuduclusterkey@0"
I20260924 07:11:14.770269 11684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-0/wal/instance:
uuid: "074aef5591034ef8ad8f6ebf7648f8f3"
format_stamp: "Formatted at 2026-09-24 07:11:14 on dist-test-slave-14ww"
server_key: "34aab3afe45c4e510e6ddf80a6b97097"
server_key_iv: "680a0fc92094b1fae4e0f4035bb79d9f"
server_key_version: "kuduclusterkey@0"
I20260924 07:11:14.771809 11684 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:11:14.782869 11699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:14.783069 11684 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:11:14.783156 11684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-0/wal
uuid: "074aef5591034ef8ad8f6ebf7648f8f3"
format_stamp: "Formatted at 2026-09-24 07:11:14 on dist-test-slave-14ww"
server_key: "34aab3afe45c4e510e6ddf80a6b97097"
server_key_iv: "680a0fc92094b1fae4e0f4035bb79d9f"
server_key_version: "kuduclusterkey@0"
I20260924 07:11:14.783236 11684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:14.816524 11684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:14.818928 11684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:14.819069 11684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:14.819321 11684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:14.819674 11684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:14.819718 11684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:14.819751 11684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:14.819767 11684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:14.827664 11684 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45873
I20260924 07:11:14.827733 11814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45873 every 8 connection(s)
I20260924 07:11:14.828074 11684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:11:14.829490 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 11684
I20260924 07:11:14.829608 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:11:14.829793 392 external_mini_cluster.cc:1428] Setting key 1e809985ce76647b2447f5aa8c935abd
Sep 24 07:11:14 dist-test-slave-14ww krb5kdc[11127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:14.840097 11817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:14.828905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44583 (local address 127.0.98.1:57587)
0924 07:11:14.829088 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:14.829130 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:14.829848 (+ 718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:14.830328 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:14.830332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:14.830628 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:14.830832 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:14.830836 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:14.831553 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:14.831554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:14.832793 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:14.832796 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:14.833356 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:14.833357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:14.833498 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:14.834186 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:14.834196 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:14.835839 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:14.837956 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:14.837963 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:14.837965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:14.838251 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:14.838475 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:14.838476 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:14.838476 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:14.838622 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:14.838930 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:14.838936 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:14.839577 (+ 641us) client_negotiation.cc:769] Sending connection context
0924 07:11:14.839706 (+ 129us) client_negotiation.cc:240] Negotiation successful
0924 07:11:14.839861 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":110,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 07:11:14.840687 11818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:14.829172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57587 (local address 127.0.98.62:44583)
0924 07:11:14.829866 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:14.829869 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:14.829882 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:11:14.830142 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:14.830143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:14.830158 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:14.830231 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:14.830925 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:14.831453 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:14.832892 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:14.833214 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:14.833284 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:14.835945 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:14.835947 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:14.835947 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:14.835954 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:14.837861 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:14.838338 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:14.838339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:14.838340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:14.838403 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:14.838702 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:14.838703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:14.838703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:14.838839 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:14.838903 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:14.840456 (+ 1553us) server_negotiation.cc:299] Negotiation successful
0924 07:11:14.840554 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":76,"threads_started":1}
I20260924 07:11:14.841259 11815 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44583
I20260924 07:11:14.841356 11815 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:14.841553 11815 heartbeater.cc:511] Master 127.0.98.62:44583 requested a full tablet report, sending...
I20260924 07:11:14.842166 11602 ts_manager.cc:194] Registered new tserver with Master: 074aef5591034ef8ad8f6ebf7648f8f3 (127.0.98.1:45873)
I20260924 07:11:14.843051 11602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57587
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 07:11:14.859949 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:57491/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44583
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:14.960527 11821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:14.960762 11821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:14.960798 11821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:14.961939 11821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:14.962000 11821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:14.962046 11821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:11:14.963449 11821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:57491/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:11:14.963693 11821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:14.963944 11821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:14.970849 11829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:14.970870 11831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:14.971102 11828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:14.971674 11821 server_base.cc:1034] running on GCE node
I20260924 07:11:14.971876 11821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:14.972172 11821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:14.973312 11821 hybrid_clock.cc:648] HybridClock initialized: now 1790233874973298 us; error 27 us; skew 500 ppm
Sep 24 07:11:14 dist-test-slave-14ww krb5kdc[11127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:14.976063 11821 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:11:14.976578 11821 webserver.cc:460] Webserver started at http://127.0.98.2:45921/ using document root <none> and password file <none>
I20260924 07:11:14.976872 11821 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:14.976933 11821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:14.977075 11821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 07:11:14 dist-test-slave-14ww krb5kdc[11127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 07:11:14 dist-test-slave-14ww krb5kdc[11127](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790233874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:14.994000 11821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-1/data/instance:
uuid: "8d6992c3f82c43aab1f9669f33f953e8"
format_stamp: "Formatted at 2026-09-24 07:11:14 on dist-test-slave-14ww"
server_key: "c0a216624441fd3b7faada3c007c775c"
server_key_iv: "17230578fd03a436bab4dc4f63d97aef"
server_key_version: "kuduclusterkey@0"
I20260924 07:11:14.994513 11821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8d6992c3f82c43aab1f9669f33f953e8"
format_stamp: "Formatted at 2026-09-24 07:11:14 on dist-test-slave-14ww"
server_key: "c0a216624441fd3b7faada3c007c775c"
server_key_iv: "17230578fd03a436bab4dc4f63d97aef"
server_key_version: "kuduclusterkey@0"
I20260924 07:11:15.003611 11821 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.001s sys 0.000s
I20260924 07:11:15.017491 11838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:15.017685 11821 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:11:15.017764 11821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-1/wal
uuid: "8d6992c3f82c43aab1f9669f33f953e8"
format_stamp: "Formatted at 2026-09-24 07:11:14 on dist-test-slave-14ww"
server_key: "c0a216624441fd3b7faada3c007c775c"
server_key_iv: "17230578fd03a436bab4dc4f63d97aef"
server_key_version: "kuduclusterkey@0"
I20260924 07:11:15.017855 11821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:15.037681 11821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:15.040182 11821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:15.040334 11821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:15.040580 11821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:15.041033 11821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:15.041077 11821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:15.041114 11821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:15.041141 11821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:15.047430 11821 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41989
I20260924 07:11:15.047838 11821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:11:15.051040 11951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41989 every 8 connection(s)
I20260924 07:11:15.053293 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 11821
I20260924 07:11:15.053403 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:11:15.053638 392 external_mini_cluster.cc:1428] Setting key ea883c486e6bd7115580f0162a565d76
Sep 24 07:11:15 dist-test-slave-14ww krb5kdc[11127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 07:11:15.064008 11818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:15.051748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33593 (local address 127.0.98.62:44583)
0924 07:11:15.051990 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:15.051992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:15.053647 (+ 1655us) server_negotiation.cc:407] Connection header received
0924 07:11:15.054417 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:15.054418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:15.054431 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:15.054461 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:15.055254 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.056045 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.057542 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.057865 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.057904 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:15.060617 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:15.060619 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:15.060619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:15.060626 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:15.062403 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:15.062900 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:15.062902 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:15.062902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:15.062961 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:15.063240 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:15.063241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:15.063241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:15.063377 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:15.063420 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:15.063791 (+ 371us) server_negotiation.cc:299] Negotiation successful
0924 07:11:15.063906 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212}
I20260924 07:11:15.064541 11954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:15.051739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44583 (local address 127.0.98.2:33593)
0924 07:11:15.053493 (+ 1754us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:15.053549 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:15.054304 (+ 755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:15.054638 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:15.054643 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:15.054893 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:15.055153 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:15.055158 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.056150 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.056151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:15.057445 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:15.057448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.057988 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.057989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:15.058120 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:15.058729 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:15.058739 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:15.060504 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:15.062500 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:15.062506 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:15.062508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:15.062785 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:15.063031 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:15.063032 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:15.063032 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:15.063162 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:15.063466 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:15.063472 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:15.063642 (+ 170us) client_negotiation.cc:769] Sending connection context
0924 07:11:15.064145 (+ 503us) client_negotiation.cc:240] Negotiation successful
0924 07:11:15.064309 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1707,"thread_start_us":90,"threads_started":1}
I20260924 07:11:15.065346 11952 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44583
I20260924 07:11:15.065440 11952 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:15.065640 11952 heartbeater.cc:511] Master 127.0.98.62:44583 requested a full tablet report, sending...
I20260924 07:11:15.066061 11602 ts_manager.cc:194] Registered new tserver with Master: 8d6992c3f82c43aab1f9669f33f953e8 (127.0.98.2:41989)
I20260924 07:11:15.066560 11602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33593
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 07:11:15.099581 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:57491/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44583
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:15.221755 11958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:15.222106 11958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:15.222189 11958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:15.224013 11958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:15.224126 11958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:15.224210 11958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:11:15.226470 11958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:57491/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:11:15.226943 11958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:15.227262 11958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:15.233713 11966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:15.234459 11963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:15.234735 11958 server_base.cc:1034] running on GCE node
W20260924 07:11:15.234975 11964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:15.235227 11958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:15.235464 11958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:15.236636 11958 hybrid_clock.cc:648] HybridClock initialized: now 1790233875236611 us; error 36 us; skew 500 ppm
Sep 24 07:11:15 dist-test-slave-14ww krb5kdc[11127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:15.239768 11958 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:11:15.240245 11958 webserver.cc:460] Webserver started at http://127.0.98.3:37273/ using document root <none> and password file <none>
I20260924 07:11:15.240500 11958 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:15.240549 11958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:15.240666 11958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 07:11:15 dist-test-slave-14ww krb5kdc[11127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 07:11:15.257623 11958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-2/data/instance:
uuid: "2ff87d610cbb45868fa56c968de4e825"
format_stamp: "Formatted at 2026-09-24 07:11:15 on dist-test-slave-14ww"
server_key: "27c0932bc94cf5f57af78594842520fe"
server_key_iv: "3b4cdde16a3b241f82378d4af10ae7c8"
server_key_version: "kuduclusterkey@0"
I20260924 07:11:15.258181 11958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2ff87d610cbb45868fa56c968de4e825"
format_stamp: "Formatted at 2026-09-24 07:11:15 on dist-test-slave-14ww"
server_key: "27c0932bc94cf5f57af78594842520fe"
server_key_iv: "3b4cdde16a3b241f82378d4af10ae7c8"
server_key_version: "kuduclusterkey@0"
I20260924 07:11:15.260840 11958 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:11:15.273519 11973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:15.273679 11958 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 07:11:15.273757 11958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-2/wal
uuid: "2ff87d610cbb45868fa56c968de4e825"
format_stamp: "Formatted at 2026-09-24 07:11:15 on dist-test-slave-14ww"
server_key: "27c0932bc94cf5f57af78594842520fe"
server_key_iv: "3b4cdde16a3b241f82378d4af10ae7c8"
server_key_version: "kuduclusterkey@0"
I20260924 07:11:15.273839 11958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:15.287323 11958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:15.290082 11958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:15.290215 11958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:15.290437 11958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:15.290759 11958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:15.290802 11958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:15.290830 11958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:15.290848 11958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:15.309441 11958 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37207
I20260924 07:11:15.309540 12086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37207 every 8 connection(s)
I20260924 07:11:15.309859 11958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:11:15.310324 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 11958
I20260924 07:11:15.310464 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:11:15.310658 392 external_mini_cluster.cc:1428] Setting key 0deab901e366dfdf50ddafbeae0f0ad4
Sep 24 07:11:15 dist-test-slave-14ww krb5kdc[11127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:15.334206 11818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:15.323542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59173 (local address 127.0.98.62:44583)
0924 07:11:15.323588 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:15.323591 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:15.324145 (+ 554us) server_negotiation.cc:407] Connection header received
0924 07:11:15.324863 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:15.324864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:15.324876 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:15.324906 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:15.325752 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.326558 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.328042 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.328338 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.328402 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:15.331025 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:15.331027 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:15.331027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:15.331034 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:15.332690 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:15.333166 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:15.333167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:15.333167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:15.333221 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:15.333495 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:15.333496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:15.333496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:15.333621 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:15.333673 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:15.334035 (+ 362us) server_negotiation.cc:299] Negotiation successful
0924 07:11:15.334111 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 07:11:15.334996 12089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:15.323565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44583 (local address 127.0.98.3:59173)
0924 07:11:15.324008 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:15.324062 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:15.324720 (+ 658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:15.325095 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:15.325100 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:15.325357 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:15.325645 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:15.325650 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.326673 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.326674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:15.327941 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:15.327944 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.328495 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.328495 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:15.328633 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:15.329280 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:15.329290 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:15.330919 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:15.332801 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:15.332807 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:15.332809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:15.333078 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:15.333287 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:15.333287 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:15.333288 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:15.333415 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:15.333706 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:15.333712 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:15.333872 (+ 160us) client_negotiation.cc:769] Sending connection context
0924 07:11:15.334621 (+ 749us) client_negotiation.cc:240] Negotiation successful
0924 07:11:15.334776 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":91,"threads_started":1}
I20260924 07:11:15.335822 12087 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44583
I20260924 07:11:15.335927 12087 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:15.336107 12087 heartbeater.cc:511] Master 127.0.98.62:44583 requested a full tablet report, sending...
I20260924 07:11:15.336464 11602 ts_manager.cc:194] Registered new tserver with Master: 2ff87d610cbb45868fa56c968de4e825 (127.0.98.3:37207)
I20260924 07:11:15.336910 11602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59173
I20260924 07:11:15.343895 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:11:15.356961 11818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:15.348555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50936 (local address 127.0.98.62:44583)
0924 07:11:15.348593 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:15.348595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:15.349328 (+ 733us) server_negotiation.cc:407] Connection header received
0924 07:11:15.349441 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:15.349442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:15.349452 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:15.349481 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:15.350256 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.351025 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.352168 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.352477 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.352516 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:15.354091 (+ 1575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:15.354093 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:15.354093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:15.354101 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:15.355785 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:15.356181 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:15.356183 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:15.356183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:15.356235 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:15.356426 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:15.356427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:15.356428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:15.356538 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:15.356653 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:15.356772 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 07:11:15.356845 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 07:11:15.358642 11602 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50936:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:11:15.360283 11602 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:11:15.373978 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:15.363432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41989 (local address 127.0.0.1:45452)
0924 07:11:15.364719 (+ 1287us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:15.364746 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:15.364793 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:15.365151 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:15.365154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:15.365156 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:15.365211 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:15.365212 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.370278 (+ 5066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.370279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:15.371692 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:15.371695 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.373683 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.373684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:15.373764 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:15.373776 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 07:11:15.373843 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 07:11:15.373901 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1274,"thread_start_us":58,"threads_started":1}
I20260924 07:11:15.374838 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:15.363842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45873 (local address 127.0.0.1:48094)
0924 07:11:15.366287 (+ 2445us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:15.366296 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:15.366339 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:15.366575 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:15.366576 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:15.366577 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:15.366620 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:15.366621 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.368027 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.368028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:15.370098 (+ 2070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:15.370101 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.374619 (+ 4518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.374620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:15.374685 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:15.374689 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:11:15.374733 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 07:11:15.374769 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2431,"thread_start_us":75,"threads_started":1}
I20260924 07:11:15.375206 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:15.364063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48094 (local address 127.0.98.1:45873)
0924 07:11:15.364898 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:15.364903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:15.366395 (+ 1492us) server_negotiation.cc:407] Connection header received
0924 07:11:15.366432 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:15.366435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:15.366450 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:15.366516 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:15.366689 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.367936 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.371789 (+ 3853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.372560 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.372632 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:15.372687 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:15.375062 (+ 2375us) server_negotiation.cc:299] Negotiation successful
0924 07:11:15.375137 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":796,"thread_start_us":50,"threads_started":1}
I20260924 07:11:15.375543 12103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:15.363628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45452 (local address 127.0.98.2:41989)
0924 07:11:15.364675 (+ 1047us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:15.364680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:15.364924 (+ 244us) server_negotiation.cc:407] Connection header received
0924 07:11:15.364969 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:15.364971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:15.364987 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:15.365059 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:15.365291 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.370203 (+ 4912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.372757 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.373505 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.373575 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:15.373649 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:15.375425 (+ 1776us) server_negotiation.cc:299] Negotiation successful
0924 07:11:15.375465 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1012,"thread_start_us":67,"threads_started":1}
I20260924 07:11:15.376885 11867 tablet_service.cc:1467] Processing CreateTablet for tablet 12f1f1daa7dc48518aad464ce07ae447 (DEFAULT_TABLE table=test-table [id=3040c141fd554f5f932766b0dcaf43d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:15.377254 11867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12f1f1daa7dc48518aad464ce07ae447. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:15.381742 11731 tablet_service.cc:1467] Processing CreateTablet for tablet 12f1f1daa7dc48518aad464ce07ae447 (DEFAULT_TABLE table=test-table [id=3040c141fd554f5f932766b0dcaf43d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:15.382074 11731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12f1f1daa7dc48518aad464ce07ae447. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:15.384886 12106 tablet_bootstrap.cc:492] T 12f1f1daa7dc48518aad464ce07ae447 P 8d6992c3f82c43aab1f9669f33f953e8: Bootstrap starting.
I20260924 07:11:15.385913 12106 tablet_bootstrap.cc:654] T 12f1f1daa7dc48518aad464ce07ae447 P 8d6992c3f82c43aab1f9669f33f953e8: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:15.386245 12106 log.cc:824] T 12f1f1daa7dc48518aad464ce07ae447 P 8d6992c3f82c43aab1f9669f33f953e8: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:15.389909 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:15.363208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37207 (local address 127.0.0.1:44776)
0924 07:11:15.364446 (+ 1238us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:15.364487 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:15.364541 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:15.385674 (+ 21133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:15.385675 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:15.385678 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:15.385719 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:15.385721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.387511 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.387513 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:15.388651 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:15.388654 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.389675 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.389676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:15.389738 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:15.389743 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:11:15.389774 (+ 31us) client_negotiation.cc:240] Negotiation successful
0924 07:11:15.389812 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1211,"thread_start_us":70,"threads_started":1}
I20260924 07:11:15.390476 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:15.363302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44776 (local address 127.0.98.3:37207)
0924 07:11:15.385494 (+ 22192us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:15.385499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:15.385510 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:11:15.385547 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:15.385549 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:15.385566 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:15.385794 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:15.385869 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.387370 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.388808 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.389572 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.390246 (+ 674us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:15.390332 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:15.390360 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 07:11:15.390399 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22156,"thread_start_us":84,"threads_started":1}
I20260924 07:11:15.391188 12021 tablet_service.cc:1467] Processing CreateTablet for tablet 12f1f1daa7dc48518aad464ce07ae447 (DEFAULT_TABLE table=test-table [id=3040c141fd554f5f932766b0dcaf43d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:11:15.391498 12021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12f1f1daa7dc48518aad464ce07ae447. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:15.392884 12107 tablet_bootstrap.cc:492] T 12f1f1daa7dc48518aad464ce07ae447 P 074aef5591034ef8ad8f6ebf7648f8f3: Bootstrap starting.
I20260924 07:11:15.393573 12107 tablet_bootstrap.cc:654] T 12f1f1daa7dc48518aad464ce07ae447 P 074aef5591034ef8ad8f6ebf7648f8f3: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:15.393865 12107 log.cc:824] T 12f1f1daa7dc48518aad464ce07ae447 P 074aef5591034ef8ad8f6ebf7648f8f3: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:15.394676 12106 tablet_bootstrap.cc:492] T 12f1f1daa7dc48518aad464ce07ae447 P 8d6992c3f82c43aab1f9669f33f953e8: No bootstrap required, opened a new log
I20260924 07:11:15.394738 12106 ts_tablet_manager.cc:1397] T 12f1f1daa7dc48518aad464ce07ae447 P 8d6992c3f82c43aab1f9669f33f953e8: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260924 07:11:15.397223 12106 raft_consensus.cc:348] T 12f1f1daa7dc48518aad464ce07ae447 P 8d6992c3f82c43aab1f9669f33f953e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074aef5591034ef8ad8f6ebf7648f8f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45873 } } peers { permanent_uuid: "2ff87d610cbb45868fa56c968de4e825" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37207 } } peers { permanent_uuid: "8d6992c3f82c43aab1f9669f33f953e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41989 } }
I20260924 07:11:15.397388 12106 raft_consensus.cc:374] T 12f1f1daa7dc48518aad464ce07ae447 P 8d6992c3f82c43aab1f9669f33f953e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:15.397423 12106 raft_consensus.cc:729] T 12f1f1daa7dc48518aad464ce07ae447 P 8d6992c3f82c43aab1f9669f33f953e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d6992c3f82c43aab1f9669f33f953e8, State: Initialized, Role: FOLLOWER
I20260924 07:11:15.397567 12106 consensus_queue.cc:260] T 12f1f1daa7dc48518aad464ce07ae447 P 8d6992c3f82c43aab1f9669f33f953e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074aef5591034ef8ad8f6ebf7648f8f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45873 } } peers { permanent_uuid: "2ff87d610cbb45868fa56c968de4e825" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37207 } } peers { permanent_uuid: "8d6992c3f82c43aab1f9669f33f953e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41989 } }
I20260924 07:11:15.397821 12106 ts_tablet_manager.cc:1428] T 12f1f1daa7dc48518aad464ce07ae447 P 8d6992c3f82c43aab1f9669f33f953e8: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 07:11:15.397940 11952 heartbeater.cc:503] Master 127.0.98.62:44583 was elected leader, sending a full tablet report...
I20260924 07:11:15.400949 12110 tablet_bootstrap.cc:492] T 12f1f1daa7dc48518aad464ce07ae447 P 2ff87d610cbb45868fa56c968de4e825: Bootstrap starting.
I20260924 07:11:15.401983 12110 tablet_bootstrap.cc:654] T 12f1f1daa7dc48518aad464ce07ae447 P 2ff87d610cbb45868fa56c968de4e825: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:15.402279 12110 log.cc:824] T 12f1f1daa7dc48518aad464ce07ae447 P 2ff87d610cbb45868fa56c968de4e825: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:15.403102 12107 tablet_bootstrap.cc:492] T 12f1f1daa7dc48518aad464ce07ae447 P 074aef5591034ef8ad8f6ebf7648f8f3: No bootstrap required, opened a new log
I20260924 07:11:15.403178 12107 ts_tablet_manager.cc:1397] T 12f1f1daa7dc48518aad464ce07ae447 P 074aef5591034ef8ad8f6ebf7648f8f3: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260924 07:11:15.405716 12107 raft_consensus.cc:348] T 12f1f1daa7dc48518aad464ce07ae447 P 074aef5591034ef8ad8f6ebf7648f8f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074aef5591034ef8ad8f6ebf7648f8f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45873 } } peers { permanent_uuid: "2ff87d610cbb45868fa56c968de4e825" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37207 } } peers { permanent_uuid: "8d6992c3f82c43aab1f9669f33f953e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41989 } }
I20260924 07:11:15.405887 12107 raft_consensus.cc:374] T 12f1f1daa7dc48518aad464ce07ae447 P 074aef5591034ef8ad8f6ebf7648f8f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:15.405933 12107 raft_consensus.cc:729] T 12f1f1daa7dc48518aad464ce07ae447 P 074aef5591034ef8ad8f6ebf7648f8f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074aef5591034ef8ad8f6ebf7648f8f3, State: Initialized, Role: FOLLOWER
I20260924 07:11:15.406081 12107 consensus_queue.cc:260] T 12f1f1daa7dc48518aad464ce07ae447 P 074aef5591034ef8ad8f6ebf7648f8f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074aef5591034ef8ad8f6ebf7648f8f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45873 } } peers { permanent_uuid: "2ff87d610cbb45868fa56c968de4e825" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37207 } } peers { permanent_uuid: "8d6992c3f82c43aab1f9669f33f953e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41989 } }
I20260924 07:11:15.406353 12107 ts_tablet_manager.cc:1428] T 12f1f1daa7dc48518aad464ce07ae447 P 074aef5591034ef8ad8f6ebf7648f8f3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 07:11:15.406440 12110 tablet_bootstrap.cc:492] T 12f1f1daa7dc48518aad464ce07ae447 P 2ff87d610cbb45868fa56c968de4e825: No bootstrap required, opened a new log
I20260924 07:11:15.406469 11815 heartbeater.cc:503] Master 127.0.98.62:44583 was elected leader, sending a full tablet report...
I20260924 07:11:15.406570 12110 ts_tablet_manager.cc:1397] T 12f1f1daa7dc48518aad464ce07ae447 P 2ff87d610cbb45868fa56c968de4e825: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260924 07:11:15.409466 12110 raft_consensus.cc:348] T 12f1f1daa7dc48518aad464ce07ae447 P 2ff87d610cbb45868fa56c968de4e825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074aef5591034ef8ad8f6ebf7648f8f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45873 } } peers { permanent_uuid: "2ff87d610cbb45868fa56c968de4e825" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37207 } } peers { permanent_uuid: "8d6992c3f82c43aab1f9669f33f953e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41989 } }
I20260924 07:11:15.409641 12110 raft_consensus.cc:374] T 12f1f1daa7dc48518aad464ce07ae447 P 2ff87d610cbb45868fa56c968de4e825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:15.409679 12110 raft_consensus.cc:729] T 12f1f1daa7dc48518aad464ce07ae447 P 2ff87d610cbb45868fa56c968de4e825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ff87d610cbb45868fa56c968de4e825, State: Initialized, Role: FOLLOWER
I20260924 07:11:15.409806 12110 consensus_queue.cc:260] T 12f1f1daa7dc48518aad464ce07ae447 P 2ff87d610cbb45868fa56c968de4e825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074aef5591034ef8ad8f6ebf7648f8f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45873 } } peers { permanent_uuid: "2ff87d610cbb45868fa56c968de4e825" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37207 } } peers { permanent_uuid: "8d6992c3f82c43aab1f9669f33f953e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41989 } }
I20260924 07:11:15.410045 12110 ts_tablet_manager.cc:1428] T 12f1f1daa7dc48518aad464ce07ae447 P 2ff87d610cbb45868fa56c968de4e825: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 07:11:15.410158 12087 heartbeater.cc:503] Master 127.0.98.62:44583 was elected leader, sending a full tablet report...
Sep 24 07:11:15 dist-test-slave-14ww krb5kdc[11127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260924 07:11:15.551846 11953 tablet.cc:2369] T 12f1f1daa7dc48518aad464ce07ae447 P 8d6992c3f82c43aab1f9669f33f953e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:11:15.561442 12088 tablet.cc:2369] T 12f1f1daa7dc48518aad464ce07ae447 P 2ff87d610cbb45868fa56c968de4e825: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:11:15.579241 11816 tablet.cc:2369] T 12f1f1daa7dc48518aad464ce07ae447 P 074aef5591034ef8ad8f6ebf7648f8f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:11:15.702018 12114 raft_consensus.cc:482] T 12f1f1daa7dc48518aad464ce07ae447 P 2ff87d610cbb45868fa56c968de4e825 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:11:15.702178 12114 raft_consensus.cc:504] T 12f1f1daa7dc48518aad464ce07ae447 P 2ff87d610cbb45868fa56c968de4e825 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074aef5591034ef8ad8f6ebf7648f8f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45873 } } peers { permanent_uuid: "2ff87d610cbb45868fa56c968de4e825" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37207 } } peers { permanent_uuid: "8d6992c3f82c43aab1f9669f33f953e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41989 } }
I20260924 07:11:15.702605 12114 leader_election.cc:288] T 12f1f1daa7dc48518aad464ce07ae447 P 2ff87d610cbb45868fa56c968de4e825 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074aef5591034ef8ad8f6ebf7648f8f3 (127.0.98.1:45873), 8d6992c3f82c43aab1f9669f33f953e8 (127.0.98.2:41989)
I20260924 07:11:15.706761 12089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:15.702810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45873 (local address 127.0.98.3:49329)
0924 07:11:15.702908 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:15.702920 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:15.702964 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:15.703466 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:15.703478 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:15.703480 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:15.703547 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:15.703549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.704443 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.704444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:15.705690 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:15.705693 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.706513 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.706514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:15.706591 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:15.706595 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:11:15.706645 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 07:11:15.706685 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 07:11:15.707480 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:15.703224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49329 (local address 127.0.98.1:45873)
0924 07:11:15.703264 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:15.703266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:15.703274 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:11:15.703301 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:15.703302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:15.703312 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:15.703368 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:15.703633 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.704340 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.705805 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.706381 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.706456 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:15.706479 (+ 23us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:15.707356 (+ 877us) server_negotiation.cc:299] Negotiation successful
0924 07:11:15.707403 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 07:11:15.707748 11769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12f1f1daa7dc48518aad464ce07ae447" candidate_uuid: "2ff87d610cbb45868fa56c968de4e825" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074aef5591034ef8ad8f6ebf7648f8f3" is_pre_election: true
I20260924 07:11:15.707899 11769 raft_consensus.cc:2454] T 12f1f1daa7dc48518aad464ce07ae447 P 074aef5591034ef8ad8f6ebf7648f8f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ff87d610cbb45868fa56c968de4e825 in term 0.
I20260924 07:11:15.708184 12103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:15.703118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55253 (local address 127.0.98.2:41989)
0924 07:11:15.703161 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:15.703163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:15.704707 (+ 1544us) server_negotiation.cc:407] Connection header received
0924 07:11:15.704836 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:15.704837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:15.704848 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:15.704887 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:11:15.705099 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.705832 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.707209 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.707777 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.707838 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:15.707848 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:15.708076 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 07:11:15.708118 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 07:11:15.708386 12115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:15.702810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41989 (local address 127.0.98.3:55253)
0924 07:11:15.704634 (+ 1824us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:15.704646 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:15.704758 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:15.704951 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:15.704952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:15.704953 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:11:15.705028 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:15.705029 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.705943 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.705944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:15.707108 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:15.707111 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.707921 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.707922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:15.707994 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:15.707997 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:11:15.708264 (+ 267us) client_negotiation.cc:240] Negotiation successful
0924 07:11:15.708304 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1804,"mutex_wait_us":11,"thread_start_us":112,"threads_started":1}
I20260924 07:11:15.708490 11976 leader_election.cc:302] T 12f1f1daa7dc48518aad464ce07ae447 P 2ff87d610cbb45868fa56c968de4e825 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074aef5591034ef8ad8f6ebf7648f8f3, 2ff87d610cbb45868fa56c968de4e825; no voters:
I20260924 07:11:15.708776 11906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12f1f1daa7dc48518aad464ce07ae447" candidate_uuid: "2ff87d610cbb45868fa56c968de4e825" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d6992c3f82c43aab1f9669f33f953e8" is_pre_election: true
I20260924 07:11:15.708925 11906 raft_consensus.cc:2454] T 12f1f1daa7dc48518aad464ce07ae447 P 8d6992c3f82c43aab1f9669f33f953e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ff87d610cbb45868fa56c968de4e825 in term 0.
I20260924 07:11:15.709369 12114 raft_consensus.cc:2781] T 12f1f1daa7dc48518aad464ce07ae447 P 2ff87d610cbb45868fa56c968de4e825 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:11:15.709432 12114 raft_consensus.cc:482] T 12f1f1daa7dc48518aad464ce07ae447 P 2ff87d610cbb45868fa56c968de4e825 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:11:15.709458 12114 raft_consensus.cc:3037] T 12f1f1daa7dc48518aad464ce07ae447 P 2ff87d610cbb45868fa56c968de4e825 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:15.710383 12114 raft_consensus.cc:504] T 12f1f1daa7dc48518aad464ce07ae447 P 2ff87d610cbb45868fa56c968de4e825 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074aef5591034ef8ad8f6ebf7648f8f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45873 } } peers { permanent_uuid: "2ff87d610cbb45868fa56c968de4e825" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37207 } } peers { permanent_uuid: "8d6992c3f82c43aab1f9669f33f953e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41989 } }
I20260924 07:11:15.710534 12114 leader_election.cc:288] T 12f1f1daa7dc48518aad464ce07ae447 P 2ff87d610cbb45868fa56c968de4e825 [CANDIDATE]: Term 1 election: Requested vote from peers 074aef5591034ef8ad8f6ebf7648f8f3 (127.0.98.1:45873), 8d6992c3f82c43aab1f9669f33f953e8 (127.0.98.2:41989)
I20260924 07:11:15.710721 11906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12f1f1daa7dc48518aad464ce07ae447" candidate_uuid: "2ff87d610cbb45868fa56c968de4e825" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d6992c3f82c43aab1f9669f33f953e8"
I20260924 07:11:15.710770 11769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12f1f1daa7dc48518aad464ce07ae447" candidate_uuid: "2ff87d610cbb45868fa56c968de4e825" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074aef5591034ef8ad8f6ebf7648f8f3"
I20260924 07:11:15.710799 11906 raft_consensus.cc:3037] T 12f1f1daa7dc48518aad464ce07ae447 P 8d6992c3f82c43aab1f9669f33f953e8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:15.710861 11769 raft_consensus.cc:3037] T 12f1f1daa7dc48518aad464ce07ae447 P 074aef5591034ef8ad8f6ebf7648f8f3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:15.711637 11906 raft_consensus.cc:2454] T 12f1f1daa7dc48518aad464ce07ae447 P 8d6992c3f82c43aab1f9669f33f953e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ff87d610cbb45868fa56c968de4e825 in term 1.
I20260924 07:11:15.711737 11769 raft_consensus.cc:2454] T 12f1f1daa7dc48518aad464ce07ae447 P 074aef5591034ef8ad8f6ebf7648f8f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ff87d610cbb45868fa56c968de4e825 in term 1.
I20260924 07:11:15.711807 11977 leader_election.cc:302] T 12f1f1daa7dc48518aad464ce07ae447 P 2ff87d610cbb45868fa56c968de4e825 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ff87d610cbb45868fa56c968de4e825, 8d6992c3f82c43aab1f9669f33f953e8; no voters:
I20260924 07:11:15.711875 12114 raft_consensus.cc:2781] T 12f1f1daa7dc48518aad464ce07ae447 P 2ff87d610cbb45868fa56c968de4e825 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:11:15.712001 12114 raft_consensus.cc:686] T 12f1f1daa7dc48518aad464ce07ae447 P 2ff87d610cbb45868fa56c968de4e825 [term 1 LEADER]: Becoming Leader. State: Replica: 2ff87d610cbb45868fa56c968de4e825, State: Running, Role: LEADER
I20260924 07:11:15.712090 12114 consensus_queue.cc:237] T 12f1f1daa7dc48518aad464ce07ae447 P 2ff87d610cbb45868fa56c968de4e825 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074aef5591034ef8ad8f6ebf7648f8f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45873 } } peers { permanent_uuid: "2ff87d610cbb45868fa56c968de4e825" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37207 } } peers { permanent_uuid: "8d6992c3f82c43aab1f9669f33f953e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41989 } }
I20260924 07:11:15.712908 11602 catalog_manager.cc:5495] T 12f1f1daa7dc48518aad464ce07ae447 P 2ff87d610cbb45868fa56c968de4e825 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ff87d610cbb45868fa56c968de4e825 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "2ff87d610cbb45868fa56c968de4e825" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074aef5591034ef8ad8f6ebf7648f8f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ff87d610cbb45868fa56c968de4e825" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37207 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d6992c3f82c43aab1f9669f33f953e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41989 } health_report { overall_health: UNKNOWN } } }
I20260924 07:11:15.722548 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:15.719478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44792 (local address 127.0.98.3:37207)
0924 07:11:15.719524 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:15.719526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:15.719560 (+ 34us) server_negotiation.cc:407] Connection header received
0924 07:11:15.719636 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:15.719637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:15.719654 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:15.719724 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:11:15.719926 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.720600 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.721657 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:15.721942 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:15.722007 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:15.722193 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:11:15.722308 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:11:15.722388 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:15.722460 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 07:11:15.722480 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 07:11:15.724658 11769 raft_consensus.cc:1261] T 12f1f1daa7dc48518aad464ce07ae447 P 074aef5591034ef8ad8f6ebf7648f8f3 [term 1 FOLLOWER]: Refusing update from remote peer 2ff87d610cbb45868fa56c968de4e825: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:11:15.725116 11906 raft_consensus.cc:1261] T 12f1f1daa7dc48518aad464ce07ae447 P 8d6992c3f82c43aab1f9669f33f953e8 [term 1 FOLLOWER]: Refusing update from remote peer 2ff87d610cbb45868fa56c968de4e825: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:11:15.725709 12116 consensus_queue.cc:1035] T 12f1f1daa7dc48518aad464ce07ae447 P 2ff87d610cbb45868fa56c968de4e825 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074aef5591034ef8ad8f6ebf7648f8f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:11:15.725898 12114 consensus_queue.cc:1035] T 12f1f1daa7dc48518aad464ce07ae447 P 2ff87d610cbb45868fa56c968de4e825 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d6992c3f82c43aab1f9669f33f953e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:11:15.730240 12121 mvcc.cc:205] Tried to move back new op lower bound from 7332797954965413888 to 7332797954917101568. Current Snapshot: MvccSnapshot[applied={T|T < 7332797954965413888}]
I20260924 07:11:15.732342 12120 mvcc.cc:205] Tried to move back new op lower bound from 7332797954965413888 to 7332797954917101568. Current Snapshot: MvccSnapshot[applied={T|T < 7332797954965413888}]
I20260924 07:11:15.734277 11601 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:11:15.734373 11601 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:11:15.735049 11601 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 498fcfb79c3c4a42a505f58654d88800: Sending DeleteTablet for 3 replicas of tablet 12f1f1daa7dc48518aad464ce07ae447
I20260924 07:11:15.735455 12021 tablet_service.cc:1514] Processing DeleteTablet for tablet 12f1f1daa7dc48518aad464ce07ae447 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44776
I20260924 07:11:15.735679 11867 tablet_service.cc:1514] Processing DeleteTablet for tablet 12f1f1daa7dc48518aad464ce07ae447 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45452
I20260924 07:11:15.735999 11731 tablet_service.cc:1514] Processing DeleteTablet for tablet 12f1f1daa7dc48518aad464ce07ae447 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:11:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48094
I20260924 07:11:15.736560 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 11684
I20260924 07:11:15.741689 12129 tablet_replica.cc:330] T 12f1f1daa7dc48518aad464ce07ae447 P 2ff87d610cbb45868fa56c968de4e825: stopping tablet replica
I20260924 07:11:15.741796 12129 raft_consensus.cc:2229] T 12f1f1daa7dc48518aad464ce07ae447 P 2ff87d610cbb45868fa56c968de4e825 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:11:15.741916 12129 raft_consensus.cc:2258] T 12f1f1daa7dc48518aad464ce07ae447 P 2ff87d610cbb45868fa56c968de4e825 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:11:15.742058 12130 tablet_replica.cc:330] T 12f1f1daa7dc48518aad464ce07ae447 P 8d6992c3f82c43aab1f9669f33f953e8: stopping tablet replica
I20260924 07:11:15.742149 12130 raft_consensus.cc:2229] T 12f1f1daa7dc48518aad464ce07ae447 P 8d6992c3f82c43aab1f9669f33f953e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:11:15.742255 12130 raft_consensus.cc:2258] T 12f1f1daa7dc48518aad464ce07ae447 P 8d6992c3f82c43aab1f9669f33f953e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:11:15.745426 12130 ts_tablet_manager.cc:1905] T 12f1f1daa7dc48518aad464ce07ae447 P 8d6992c3f82c43aab1f9669f33f953e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:11:15.747393 12130 ts_tablet_manager.cc:1918] T 12f1f1daa7dc48518aad464ce07ae447 P 8d6992c3f82c43aab1f9669f33f953e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:11:15.747526 12130 log.cc:1196] T 12f1f1daa7dc48518aad464ce07ae447 P 8d6992c3f82c43aab1f9669f33f953e8: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-1/wal/wals/12f1f1daa7dc48518aad464ce07ae447
W20260924 07:11:15.747947 11589 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45873 (error 108)
I20260924 07:11:15.748301 12130 ts_tablet_manager.cc:1939] T 12f1f1daa7dc48518aad464ce07ae447 P 8d6992c3f82c43aab1f9669f33f953e8: Deleting consensus metadata
I20260924 07:11:15.748795 11590 catalog_manager.cc:4841] TS 8d6992c3f82c43aab1f9669f33f953e8 (127.0.98.2:41989): tablet 12f1f1daa7dc48518aad464ce07ae447 (table test-table [id=3040c141fd554f5f932766b0dcaf43d0]) successfully deleted
I20260924 07:11:15.749009 12129 ts_tablet_manager.cc:1905] T 12f1f1daa7dc48518aad464ce07ae447 P 2ff87d610cbb45868fa56c968de4e825: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:11:15.750954 12129 ts_tablet_manager.cc:1918] T 12f1f1daa7dc48518aad464ce07ae447 P 2ff87d610cbb45868fa56c968de4e825: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:11:15.751057 12129 log.cc:1196] T 12f1f1daa7dc48518aad464ce07ae447 P 2ff87d610cbb45868fa56c968de4e825: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790233772407728-392-0/minicluster-data/ts-2/wal/wals/12f1f1daa7dc48518aad464ce07ae447
I20260924 07:11:15.751322 12129 ts_tablet_manager.cc:1939] T 12f1f1daa7dc48518aad464ce07ae447 P 2ff87d610cbb45868fa56c968de4e825: Deleting consensus metadata
I20260924 07:11:15.752797 11588 catalog_manager.cc:4841] TS 2ff87d610cbb45868fa56c968de4e825 (127.0.98.3:37207): tablet 12f1f1daa7dc48518aad464ce07ae447 (table test-table [id=3040c141fd554f5f932766b0dcaf43d0]) successfully deleted
I20260924 07:11:15.753463 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 11821
I20260924 07:11:15.755085 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:15.754202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45873 (local address 127.0.0.1:48096)
0924 07:11:15.755011 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:15.755054 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":783}
W20260924 07:11:15.756268 11589 catalog_manager.cc:4584] TS 074aef5591034ef8ad8f6ebf7648f8f3 (127.0.98.1:45873): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 12f1f1daa7dc48518aad464ce07ae447: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45873: connect: Connection refused (error 111)
I20260924 07:11:15.760664 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 11958
I20260924 07:11:15.766149 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 11571
I20260924 07:11:15.770552 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 07:11:16.114887 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 07:11:16.114979 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 07:11:16.458936 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 07:11:16.459 UTC [11157] LOG: received smart shutdown request
2026-09-24 07:11:16.462 UTC [11157] LOG: background worker "logical replication launcher" (PID 11166) exited with exit code 1
2026-09-24 07:11:16.462 UTC [11161] LOG: shutting down
2026-09-24 07:11:16.637 UTC [11157] LOG: database system is shut down
2026-09-24T07:11:16Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (44876 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:11:16 dist-test-slave-14ww krb5kdc[12141](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:11:16 dist-test-slave-14ww krb5kdc[12141](info): set up 2 sockets
Sep 24 07:11:16 dist-test-slave-14ww krb5kdc[12141](info): commencing operation
krb5kdc: starting...
W20260924 07:11:17.784559 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:11:17 dist-test-slave-14ww krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:11:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:11:17Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 07:11:17.951422 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskIclK1b/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/postgres -l logfile start
2026-09-24 07:11:20.474 UTC [12171] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 07:11:20.474 UTC [12171] LOG: listening on IPv4 address "127.0.98.20", port 33327
2026-09-24 07:11:20.478 UTC [12171] LOG: listening on Unix socket "/tmp/.s.PGSQL.33327"
2026-09-24 07:11:20.513 UTC [12174] LOG: database system was shut down at 2026-09-24 07:11:18 UTC
2026-09-24 07:11:20.517 UTC [12171] LOG: database system is ready to accept connections
I20260924 07:11:21.456549 392 mini_postgres.cc:96] Postgres bound to 33327
2026-09-24 07:11:21.462 UTC [12182] FATAL: database "slave" does not exist
127.0.98.20:33327 - accepting connections
I20260924 07:11:21.462869 392 mini_ranger.cc:161] Starting Ranger...
I20260924 07:11:21.476676 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 07:11:21.978747 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 07:11:21.978835 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ranger-admin
2026-09-24 07:11:22,271 [I] DB FLAVOR :POSTGRES
2026-09-24 07:11:22,271 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 07:11:22,271 [I] Checking connection..
2026-09-24 07:11:22,271 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 07:11:22,514 [I] Checking connection passed.
2026-09-24 07:11:22,514 [I] --------- Verifying version history table ---------
2026-09-24 07:11:22,514 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 07:11:22,767 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 07:11:22,768 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 07:11:23,011 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 07:11:23,011 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 07:11:23,011 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33327/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 07:11:23.241 UTC [12246] WARNING: there is no transaction in progress
2026-09-24 07:11:23,248 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 07:11:23,248 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 07:11:23,515 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 07:11:23,515 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 07:11:23,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 07:11:23,754 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 07:11:23,997 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-14ww.c.gcp-upstream.internal','N') ;"
2026-09-24 07:11:24,240 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 07:11:24,483 [I] Table x_portal_user does not exist in database ranger
2026-09-24 07:11:24,483 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 07:11:24,729 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 07:11:24,729 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 07:11:24,974 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 07:11:25,229 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 07:11:25,229 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33327/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 07:11:26.220 UTC [12354] WARNING: there is no transaction in progress
2026-09-24 07:11:26.237 UTC [12354] WARNING: there is no transaction in progress
2026-09-24 07:11:26.253 UTC [12354] WARNING: there is no transaction in progress
2026-09-24 07:11:26.267 UTC [12354] WARNING: there is no transaction in progress
2026-09-24 07:11:26.286 UTC [12354] WARNING: there is no transaction in progress
2026-09-24 07:11:26.305 UTC [12354] WARNING: there is no transaction in progress
2026-09-24 07:11:26.370 UTC [12354] WARNING: there is no transaction in progress
2026-09-24 07:11:26.377 UTC [12354] WARNING: there is no transaction in progress
2026-09-24 07:11:26.391 UTC [12354] WARNING: there is no transaction in progress
2026-09-24 07:11:26.401 UTC [12354] WARNING: there is no transaction in progress
2026-09-24 07:11:26.411 UTC [12354] WARNING: there is no transaction in progress
2026-09-24 07:11:26.425 UTC [12354] WARNING: there is no transaction in progress
2026-09-24 07:11:26.435 UTC [12354] WARNING: there is no transaction in progress
2026-09-24 07:11:26.441 UTC [12354] WARNING: there is no transaction in progress
2026-09-24 07:11:26.447 UTC [12354] WARNING: there is no transaction in progress
2026-09-24 07:11:26.454 UTC [12354] WARNING: there is no transaction in progress
2026-09-24 07:11:27,099 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 07:11:27,099 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 07:11:27,344 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 07:11:27,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 07:11:27,593 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 07:11:27,593 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 07:11:27,849 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 07:11:27,850 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 07:11:28,106 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 07:11:28,330 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-14ww.c.gcp-upstream.internal';"
2026-09-24 07:11:28,574 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 07:11:28,574 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33327/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 07:11:28,824 [I] DB_PATCHES have already been applied
I20260924 07:11:28.830385 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskIclK1b/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 07:11:28.830466 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 07:11:28.833508 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:34715
Sep 24, 2026 7:11:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:11:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:11:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 7:11:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 7:11:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 7:11:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 7:11:29 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 07:11:29.971909 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.509s user 0.000s sys 0.008s
Sep 24, 2026 7:11:48 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 7:11:48 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 7:11:48 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 7:11:48 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 7:11:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 7:11:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 7:11:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 7:11:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 7:11:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 7:11:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 7:11:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 7:11:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 7:11:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 7:11:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 7:11:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 7:11:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 7:11:48 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 7:11:49 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 07:11:50.398774 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 07:11:50 dist-test-slave-14ww krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233910, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 07:11:50.479568 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 07:11:50.488473 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 07:11:51.406848 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 07:11:51.406937 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ranger-kms
I20260924 07:11:51.406960 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:33327
2026-09-24 07:11:51,564 [I] DB FLAVOR :POSTGRES
2026-09-24 07:11:51,564 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 07:11:51,564 [I] Checking connection
2026-09-24 07:11:51,564 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33327/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 07:11:51,819 [I] connection success
2026-09-24 07:11:51,820 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 07:11:51,820 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 07:11:51,820 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33327/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 07:11:52,063 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 07:11:52,063 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 07:11:52,064 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 07:11:52,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:33327/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 07:11:52,308 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 07:11:52.741811 392 mini_ranger_kms.cc:310] Created kms service
I20260924 07:11:52.911818 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 07:11:52.953514 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 07:11:52 dist-test-slave-14ww krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233910, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 07:11:53.332309 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 07:11:53.332438 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskIclK1b/build/release/bin/postgresql.jar:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskIclK1b/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 07:11:53.332458 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 07:11:53.336134 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12534
I20260924 07:11:53.336230 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:56931
Sep 24, 2026 7:11:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:11:54 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:11:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 7:11:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 7:11:54 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 7:11:54 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 07:11:55.573163 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.094s user 0.000s sys 0.142s
I20260924 07:11:55.573257 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 07:11:55.573302 392 mini_ranger_kms.cc:401] 127.0.98.20:56931/kms/v1/keys
Sep 24 07:11:55 dist-test-slave-14ww krb5kdc[12141](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790233915, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 07:11:56 dist-test-slave-14ww krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233915, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 07:11:56 dist-test-slave-14ww krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233915, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 07:11:58 dist-test-slave-14ww krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:11:58.660076 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38983
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38983
--ranger_config_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56931/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:58.740094 12585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:58.740278 12585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:58.740300 12585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:58.741353 12585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:11:58.741389 12585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:58.741403 12585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:58.741415 12585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:11:58.741433 12585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:11:58.742875 12585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:56931/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38983
--ranger_config_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:11:58.743043 12585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:58.743189 12585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:58.748050 12593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:58.748099 12585 server_base.cc:1034] running on GCE node
W20260924 07:11:58.748202 12591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:58.748346 12590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:58.748602 12585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:58.748910 12585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:58.750064 12585 hybrid_clock.cc:648] HybridClock initialized: now 1790233918750059 us; error 39 us; skew 500 ppm
Sep 24 07:11:58 dist-test-slave-14ww krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:58.764000 12585 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:11:58.764487 12585 webserver.cc:460] Webserver started at http://127.0.98.62:44449/ using document root <none> and password file <none>
I20260924 07:11:58.764667 12585 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:58.764703 12585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:58.764851 12585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 07:11:58 dist-test-slave-14ww krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 07:11:58.817163 12585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "02b1526aa9704241b56acb1aa58388ab"
format_stamp: "Formatted at 2026-09-24 07:11:58 on dist-test-slave-14ww"
server_key: "ed9e4ba2d94125b80401fc0467806cf9"
server_key_iv: "2b2e1a0b720e579bdc810fcdbdf652ee"
server_key_version: "kuduclusterkey@0"
I20260924 07:11:58.817574 12585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "02b1526aa9704241b56acb1aa58388ab"
format_stamp: "Formatted at 2026-09-24 07:11:58 on dist-test-slave-14ww"
server_key: "ed9e4ba2d94125b80401fc0467806cf9"
server_key_iv: "2b2e1a0b720e579bdc810fcdbdf652ee"
server_key_version: "kuduclusterkey@0"
I20260924 07:11:58.819010 12585 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:11:58.829631 12600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:58.829849 12585 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:11:58.830160 12585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "02b1526aa9704241b56acb1aa58388ab"
format_stamp: "Formatted at 2026-09-24 07:11:58 on dist-test-slave-14ww"
server_key: "ed9e4ba2d94125b80401fc0467806cf9"
server_key_iv: "2b2e1a0b720e579bdc810fcdbdf652ee"
server_key_version: "kuduclusterkey@0"
I20260924 07:11:58.830266 12585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:58.843140 12585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:58.845135 12585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:58.845283 12585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:58.850055 12585 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38983
I20260924 07:11:58.850134 12652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38983 every 8 connection(s)
I20260924 07:11:58.850458 12585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:11:58.851120 12653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:11:58.853343 12653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab: Bootstrap starting.
I20260924 07:11:58.853885 12653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab: Neither blocks nor log segments found. Creating new log.
I20260924 07:11:58.854174 12653 log.cc:824] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab: Log is configured to *not* fsync() on all Append() calls
I20260924 07:11:58.854808 12653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab: No bootstrap required, opened a new log
I20260924 07:11:58.855365 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 12585
I20260924 07:11:58.855472 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:11:58.855667 392 external_mini_cluster.cc:1428] Setting key c7b46188f36b0f922e2bd62e4daa46d3
I20260924 07:11:58.857276 12653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b1526aa9704241b56acb1aa58388ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38983 } }
I20260924 07:11:58.857429 12653 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:11:58.857491 12653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02b1526aa9704241b56acb1aa58388ab, State: Initialized, Role: FOLLOWER
I20260924 07:11:58.857628 12653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b1526aa9704241b56acb1aa58388ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38983 } }
I20260924 07:11:58.857728 12653 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:11:58.857762 12653 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:11:58.857836 12653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:11:58.858531 12653 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b1526aa9704241b56acb1aa58388ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38983 } }
I20260924 07:11:58.858652 12653 leader_election.cc:302] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02b1526aa9704241b56acb1aa58388ab; no voters:
I20260924 07:11:58.858856 12653 leader_election.cc:288] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:11:58.858882 12658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:11:58.859104 12658 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab [term 1 LEADER]: Becoming Leader. State: Replica: 02b1526aa9704241b56acb1aa58388ab, State: Running, Role: LEADER
I20260924 07:11:58.859211 12653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:11:58.859228 12658 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b1526aa9704241b56acb1aa58388ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38983 } }
I20260924 07:11:58.859556 12658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02b1526aa9704241b56acb1aa58388ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b1526aa9704241b56acb1aa58388ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38983 } } }
I20260924 07:11:58.859623 12658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:58.859717 12658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02b1526aa9704241b56acb1aa58388ab. Latest consensus state: current_term: 1 leader_uuid: "02b1526aa9704241b56acb1aa58388ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b1526aa9704241b56acb1aa58388ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38983 } } }
I20260924 07:11:58.859748 12658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab [sys.catalog]: This master's current role is: LEADER
I20260924 07:11:58.859961 12662 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:11:58.861157 12653 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 07:11:58.861708 12662 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 07:11:58 dist-test-slave-14ww krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:58.867369 12662 catalog_manager.cc:1347] Generated new cluster ID: 9dd2cb1e9e6b4879a2d138d5e4ce2160
I20260924 07:11:58.867506 12662 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:11:58.870487 12657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:58.856162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57790 (local address 127.0.98.62:38983)
0924 07:11:58.856902 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:58.856911 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:58.856952 (+ 41us) server_negotiation.cc:407] Connection header received
0924 07:11:58.857279 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:58.857287 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:58.857344 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:58.857486 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:58.858351 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:58.859999 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:58.861161 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:58.861778 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:58.861896 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:58.865056 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:58.865099 (+ 43us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:58.865100 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:58.865212 (+ 112us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:58.868147 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:58.868537 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:58.868540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:58.868540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:58.868665 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:58.868941 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:58.868942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:58.868943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:58.869285 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:58.869449 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:58.869870 (+ 421us) server_negotiation.cc:299] Negotiation successful
0924 07:11:58.870200 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":63,"threads_started":1}
I20260924 07:11:58.881971 12662 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:11:58.882669 12662 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:11:58.891211 12662 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab: Generated new TSK 0
I20260924 07:11:58.891453 12662 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 07:11:59.561275 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56931/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38983
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:11:59.646129 12698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:59.646353 12698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:59.646387 12698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:59.647497 12698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:59.647562 12698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:59.647605 12698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:11:59.649367 12698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:56931/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:11:59.649619 12698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:59.649847 12698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:59.654716 12706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:59.654861 12703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:59.654685 12704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:59.655753 12698 server_base.cc:1034] running on GCE node
I20260924 07:11:59.655944 12698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:59.656183 12698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:59.657342 12698 hybrid_clock.cc:648] HybridClock initialized: now 1790233919657322 us; error 33 us; skew 500 ppm
Sep 24 07:11:59 dist-test-slave-14ww krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:59.660053 12698 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:11:59.660568 12698 webserver.cc:460] Webserver started at http://127.0.98.1:35947/ using document root <none> and password file <none>
I20260924 07:11:59.660852 12698 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:59.660921 12698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:59.661063 12698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 07:11:59 dist-test-slave-14ww krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 07:11:59.671850 12698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-0/data/instance:
uuid: "0b80daed62c048bf932ad2cda642ac03"
format_stamp: "Formatted at 2026-09-24 07:11:59 on dist-test-slave-14ww"
server_key: "047dd54772f1128627bc9204c577c6de"
server_key_iv: "76dcba85dd9609b49fab256ce277fc4c"
server_key_version: "kuduclusterkey@0"
I20260924 07:11:59.672281 12698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0b80daed62c048bf932ad2cda642ac03"
format_stamp: "Formatted at 2026-09-24 07:11:59 on dist-test-slave-14ww"
server_key: "047dd54772f1128627bc9204c577c6de"
server_key_iv: "76dcba85dd9609b49fab256ce277fc4c"
server_key_version: "kuduclusterkey@0"
I20260924 07:11:59.674104 12698 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260924 07:11:59.686905 12713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:59.687412 12698 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:11:59.687507 12698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-0/wal
uuid: "0b80daed62c048bf932ad2cda642ac03"
format_stamp: "Formatted at 2026-09-24 07:11:59 on dist-test-slave-14ww"
server_key: "047dd54772f1128627bc9204c577c6de"
server_key_iv: "76dcba85dd9609b49fab256ce277fc4c"
server_key_version: "kuduclusterkey@0"
I20260924 07:11:59.687580 12698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:59.711223 12698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:59.714257 12698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:59.714453 12698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:59.714735 12698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:59.715152 12698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:59.715248 12698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:59.715350 12698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:59.715438 12698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:59.722259 12698 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45281
I20260924 07:11:59.722744 12698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:11:59.726011 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 12698
I20260924 07:11:59.726112 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:11:59.726326 392 external_mini_cluster.cc:1428] Setting key 2e57ff6d58db38ac0d96b82eef5decf4
I20260924 07:11:59.724617 12828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45281 every 8 connection(s)
Sep 24 07:11:59 dist-test-slave-14ww krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:11:59.737093 12832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:59.725530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53641 (local address 127.0.98.62:38983)
0924 07:11:59.726232 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0924 07:11:59.726234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:11:59.726246 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:11:59.727363 (+ 1117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:11:59.727364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:11:59.727378 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:11:59.727444 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:11:59.728225 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:59.729039 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:59.730522 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:11:59.730825 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:11:59.730890 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:59.733462 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:11:59.733464 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:11:59.733464 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:11:59.733472 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:11:59.735413 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:59.735914 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:59.735915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:59.735916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:59.735984 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:11:59.736305 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:11:59.736306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:11:59.736306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:11:59.736440 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:11:59.736483 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 07:11:59.736893 (+ 410us) server_negotiation.cc:299] Negotiation successful
0924 07:11:59.736979 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":76,"threads_started":1}
I20260924 07:11:59.737789 12831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:11:59.725236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38983 (local address 127.0.98.1:53641)
0924 07:11:59.725802 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0924 07:11:59.725848 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 07:11:59.727203 (+ 1355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:11:59.727568 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:11:59.727572 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:11:59.727816 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:11:59.728111 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:59.728117 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:59.729143 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:59.729144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:59.730427 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:11:59.730430 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:11:59.730971 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:11:59.730972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:11:59.731131 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:11:59.731729 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:11:59.731741 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:11:59.733354 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:11:59.735491 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:59.735498 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:59.735500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:59.735811 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:59.736077 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:11:59.736078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:11:59.736079 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:11:59.736219 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:11:59.736532 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:11:59.736538 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:11:59.736705 (+ 167us) client_negotiation.cc:769] Sending connection context
0924 07:11:59.737414 (+ 709us) client_negotiation.cc:240] Negotiation successful
0924 07:11:59.737571 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":80,"threads_started":1}
I20260924 07:11:59.738520 12829 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38983
I20260924 07:11:59.738610 12829 heartbeater.cc:464] Registering TS with master...
I20260924 07:11:59.738781 12829 heartbeater.cc:511] Master 127.0.98.62:38983 requested a full tablet report, sending...
I20260924 07:11:59.739308 12617 ts_manager.cc:194] Registered new tserver with Master: 0b80daed62c048bf932ad2cda642ac03 (127.0.98.1:45281)
I20260924 07:11:59.740211 12617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53641
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 07:11:59.767432 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56931/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38983
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:11:59 dist-test-slave-14ww krb5kdc[12141](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790233919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 07:11:59.898244 12835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:11:59.898500 12835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:11:59.898536 12835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:11:59.900290 12835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:11:59.900348 12835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:11:59.900394 12835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:11:59.902554 12835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:56931/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:11:59.902786 12835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:11:59.903072 12835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:11:59.909372 12843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:59.909384 12842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:11:59.912374 12845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:11:59.912672 12835 server_base.cc:1034] running on GCE node
I20260924 07:11:59.912905 12835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:11:59.913172 12835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:11:59.916436 12835 hybrid_clock.cc:648] HybridClock initialized: now 1790233919916421 us; error 28 us; skew 500 ppm
Sep 24 07:11:59 dist-test-slave-14ww krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:11:59.919348 12835 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:11:59.919906 12835 webserver.cc:460] Webserver started at http://127.0.98.2:38313/ using document root <none> and password file <none>
I20260924 07:11:59.920238 12835 fs_manager.cc:359] Metadata directory not provided
I20260924 07:11:59.920310 12835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:11:59.920511 12835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 07:11:59 dist-test-slave-14ww krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 07:11:59.938452 12835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-1/data/instance:
uuid: "f41e19b9881f4e06bed642d711b719b7"
format_stamp: "Formatted at 2026-09-24 07:11:59 on dist-test-slave-14ww"
server_key: "3a119271f5ecea765a877db881536ab0"
server_key_iv: "14d27984c69d4bbb3e621031d58b2a97"
server_key_version: "kuduclusterkey@0"
I20260924 07:11:59.938979 12835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f41e19b9881f4e06bed642d711b719b7"
format_stamp: "Formatted at 2026-09-24 07:11:59 on dist-test-slave-14ww"
server_key: "3a119271f5ecea765a877db881536ab0"
server_key_iv: "14d27984c69d4bbb3e621031d58b2a97"
server_key_version: "kuduclusterkey@0"
I20260924 07:11:59.940568 12835 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:11:59.961158 12852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:59.961403 12835 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.001s
I20260924 07:11:59.961493 12835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-1/wal
uuid: "f41e19b9881f4e06bed642d711b719b7"
format_stamp: "Formatted at 2026-09-24 07:11:59 on dist-test-slave-14ww"
server_key: "3a119271f5ecea765a877db881536ab0"
server_key_iv: "14d27984c69d4bbb3e621031d58b2a97"
server_key_version: "kuduclusterkey@0"
I20260924 07:11:59.961580 12835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:11:59.986021 12835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:11:59.988425 12835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:11:59.988613 12835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:11:59.988895 12835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:11:59.989295 12835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:11:59.989380 12835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:59.989449 12835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:11:59.989507 12835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:11:59.998375 12835 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43167
I20260924 07:11:59.998834 12835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:12:00.002007 12966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43167 every 8 connection(s)
I20260924 07:12:00.004683 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 12835
I20260924 07:12:00.004815 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:12:00.005004 392 external_mini_cluster.cc:1428] Setting key 103bb85bdfc6c05c70ad5792ab79409a
Sep 24 07:12:00 dist-test-slave-14ww krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:12:00.013497 12832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:00.002621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59407 (local address 127.0.98.62:38983)
0924 07:12:00.002847 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:00.002850 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:00.003049 (+ 199us) server_negotiation.cc:407] Connection header received
0924 07:12:00.003767 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:00.003769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:00.003781 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:00.003836 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:00.004389 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:00.005215 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:00.006672 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:00.007018 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:00.007061 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:00.009820 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:00.009822 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:00.009822 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:00.009830 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:00.011752 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:00.012281 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:00.012282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:00.012283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:00.012341 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:00.012644 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:00.012645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:00.012646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:00.012800 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:00.012847 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:00.013251 (+ 404us) server_negotiation.cc:299] Negotiation successful
0924 07:12:00.013345 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191}
I20260924 07:12:00.014302 12969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:00.002607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38983 (local address 127.0.98.2:59407)
0924 07:12:00.002912 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:00.002955 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:00.003653 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:00.003908 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:00.003912 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:00.004102 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:00.004289 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:00.004293 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:00.005311 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:00.005313 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:00.006576 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:00.006579 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:00.007160 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:00.007161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:00.007308 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:00.007888 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:00.007899 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:00.009712 (+ 1813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:12:00.011858 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:00.011864 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:00.011867 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:00.012187 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:00.012415 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:00.012416 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:00.012416 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:00.012565 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:00.012899 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:12:00.012905 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:12:00.013082 (+ 177us) client_negotiation.cc:769] Sending connection context
0924 07:12:00.013857 (+ 775us) client_negotiation.cc:240] Negotiation successful
0924 07:12:00.014023 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 07:12:00.014981 12967 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38983
I20260924 07:12:00.015072 12967 heartbeater.cc:464] Registering TS with master...
I20260924 07:12:00.015255 12967 heartbeater.cc:511] Master 127.0.98.62:38983 requested a full tablet report, sending...
I20260924 07:12:00.015731 12617 ts_manager.cc:194] Registered new tserver with Master: f41e19b9881f4e06bed642d711b719b7 (127.0.98.2:43167)
I20260924 07:12:00.016189 12617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59407
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 07:12:00.035424 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56931/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38983
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:12:00.155387 12972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:00.155726 12972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:00.155803 12972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:00.167122 12972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:00.167212 12972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:00.167256 12972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:12:00.168881 12972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:56931/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:12:00.169142 12972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:00.169415 12972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:00.174355 12977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:00.174518 12978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:00.174736 12980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:00.174980 12972 server_base.cc:1034] running on GCE node
I20260924 07:12:00.175169 12972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:00.175410 12972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:00.176576 12972 hybrid_clock.cc:648] HybridClock initialized: now 1790233920176565 us; error 25 us; skew 500 ppm
Sep 24 07:12:00 dist-test-slave-14ww krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:12:00.179117 12972 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:12:00.179585 12972 webserver.cc:460] Webserver started at http://127.0.98.3:44529/ using document root <none> and password file <none>
I20260924 07:12:00.179836 12972 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:00.179911 12972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:00.180043 12972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 07:12:00 dist-test-slave-14ww krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 07:12:00.192795 12972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-2/data/instance:
uuid: "0041dfc393244a119e6a7a1ce733b349"
format_stamp: "Formatted at 2026-09-24 07:12:00 on dist-test-slave-14ww"
server_key: "653b8d3f5e75b723eec09a43cf3b3d19"
server_key_iv: "4da447bd65268d1d6004b37d614eeb80"
server_key_version: "kuduclusterkey@0"
I20260924 07:12:00.193346 12972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0041dfc393244a119e6a7a1ce733b349"
format_stamp: "Formatted at 2026-09-24 07:12:00 on dist-test-slave-14ww"
server_key: "653b8d3f5e75b723eec09a43cf3b3d19"
server_key_iv: "4da447bd65268d1d6004b37d614eeb80"
server_key_version: "kuduclusterkey@0"
I20260924 07:12:00.214480 12972 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.001s sys 0.001s
I20260924 07:12:00.233850 12987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:00.234125 12972 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.000s sys 0.000s
I20260924 07:12:00.234234 12972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-2/wal
uuid: "0041dfc393244a119e6a7a1ce733b349"
format_stamp: "Formatted at 2026-09-24 07:12:00 on dist-test-slave-14ww"
server_key: "653b8d3f5e75b723eec09a43cf3b3d19"
server_key_iv: "4da447bd65268d1d6004b37d614eeb80"
server_key_version: "kuduclusterkey@0"
I20260924 07:12:00.234339 12972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:00.254367 12972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:00.256870 12972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:00.257077 12972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:00.257326 12972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:12:00.257733 12972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:12:00.257835 12972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:00.257912 12972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:12:00.257973 12972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:00.264606 12972 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37909
I20260924 07:12:00.265228 12972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:12:00.267949 13100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37909 every 8 connection(s)
I20260924 07:12:00.270689 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 12972
I20260924 07:12:00.270800 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:12:00.271004 392 external_mini_cluster.cc:1428] Setting key 4f11a715745f9d09c4eab069e5111733
Sep 24 07:12:00 dist-test-slave-14ww krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:12:00.279713 12832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:00.268821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44083 (local address 127.0.98.62:38983)
0924 07:12:00.268865 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:00.268867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:00.269359 (+ 492us) server_negotiation.cc:407] Connection header received
0924 07:12:00.270081 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:00.270082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:00.270095 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:00.270126 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:00.271137 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:00.271898 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:00.273364 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:00.273677 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:00.273719 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:00.276306 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:00.276307 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:00.276307 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:00.276314 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:00.278133 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:00.278632 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:00.278634 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:00.278634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:00.278692 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:00.278973 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:00.278974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:00.278974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:00.279105 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:00.279148 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:00.279535 (+ 387us) server_negotiation.cc:299] Negotiation successful
0924 07:12:00.279617 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 07:12:00.280474 13103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:00.268508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38983 (local address 127.0.98.3:44083)
0924 07:12:00.269233 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:00.269273 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:00.269951 (+ 678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:00.270315 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:00.270320 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:00.270580 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:00.270972 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:00.270979 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:00.272018 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:00.272019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:00.273260 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:00.273263 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:00.273818 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:00.273819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:00.273950 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:00.274581 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:00.274591 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:00.276195 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:12:00.278229 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:00.278235 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:00.278237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:00.278536 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:00.278766 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:00.278767 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:00.278767 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:00.278893 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:00.279196 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:12:00.279201 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:12:00.279367 (+ 166us) client_negotiation.cc:769] Sending connection context
0924 07:12:00.280083 (+ 716us) client_negotiation.cc:240] Negotiation successful
0924 07:12:00.280244 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":78,"threads_started":1}
I20260924 07:12:00.281142 13101 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38983
I20260924 07:12:00.281234 13101 heartbeater.cc:464] Registering TS with master...
I20260924 07:12:00.281425 13101 heartbeater.cc:511] Master 127.0.98.62:38983 requested a full tablet report, sending...
I20260924 07:12:00.281826 12617 ts_manager.cc:194] Registered new tserver with Master: 0041dfc393244a119e6a7a1ce733b349 (127.0.98.3:37909)
I20260924 07:12:00.282303 12617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44083
I20260924 07:12:00.283627 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:12:00.284350 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 12698
I20260924 07:12:00.288460 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 12835
I20260924 07:12:00.294713 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 12972
I20260924 07:12:00.301059 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 12585
I20260924 07:12:00.305729 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38983
--webserver_interface=127.0.98.62
--webserver_port=44449
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38983
--ranger_config_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56931/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56931/kms with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:12:00.386768 13104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:00.386991 13104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:00.387028 13104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:00.388144 13104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:12:00.388211 13104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:00.388236 13104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:00.388265 13104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:12:00.388288 13104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:12:00.389899 13104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56931/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38983
--ranger_config_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:12:00.390152 13104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:00.390364 13104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:00.396145 13109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:00.396157 13110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:00.396193 13112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:00.396665 13104 server_base.cc:1034] running on GCE node
I20260924 07:12:00.396903 13104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:00.397166 13104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:00.398303 13104 hybrid_clock.cc:648] HybridClock initialized: now 1790233920398285 us; error 32 us; skew 500 ppm
Sep 24 07:12:00 dist-test-slave-14ww krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:12:00.400556 13104 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:12:00.401038 13104 webserver.cc:460] Webserver started at http://127.0.98.62:44449/ using document root <none> and password file <none>
I20260924 07:12:00.401247 13104 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:00.401321 13104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 07:12:00 dist-test-slave-14ww krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 07:12:00.458799 13104 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:12:00.459507 13120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:00.459687 13104 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:12:00.459789 13104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "02b1526aa9704241b56acb1aa58388ab"
format_stamp: "Formatted at 2026-09-24 07:11:58 on dist-test-slave-14ww"
server_key: "ed9e4ba2d94125b80401fc0467806cf9"
server_key_iv: "2b2e1a0b720e579bdc810fcdbdf652ee"
server_key_version: "kuduclusterkey@0"
I20260924 07:12:00.460176 13104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:00.481088 13104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:00.482712 13104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:00.482877 13104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:00.487111 13104 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38983
I20260924 07:12:00.487157 13172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38983 every 8 connection(s)
I20260924 07:12:00.487509 13104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:12:00.488651 13173 sys_catalog.cc:263] Verifying existing consensus state
I20260924 07:12:00.489313 13173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab: Bootstrap starting.
I20260924 07:12:00.491933 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 13104
I20260924 07:12:00.492206 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56931/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45281
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38983
--webserver_port=35947
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56931/kms with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:12:00.497252 13173 log.cc:824] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab: Log is configured to *not* fsync() on all Append() calls
I20260924 07:12:00.498843 13173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:12:00.499116 13173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab: Bootstrap complete.
I20260924 07:12:00.501530 13173 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b1526aa9704241b56acb1aa58388ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38983 } }
I20260924 07:12:00.501771 13173 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02b1526aa9704241b56acb1aa58388ab, State: Initialized, Role: FOLLOWER
I20260924 07:12:00.501964 13173 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b1526aa9704241b56acb1aa58388ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38983 } }
I20260924 07:12:00.502059 13173 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:12:00.502110 13173 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:12:00.502157 13173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab [term 1 FOLLOWER]: Advancing to term 2
I20260924 07:12:00.503082 13173 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b1526aa9704241b56acb1aa58388ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38983 } }
I20260924 07:12:00.503201 13173 leader_election.cc:302] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02b1526aa9704241b56acb1aa58388ab; no voters:
I20260924 07:12:00.503386 13173 leader_election.cc:288] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 07:12:00.503419 13177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab [term 2 FOLLOWER]: Leader election won for term 2
I20260924 07:12:00.503518 13177 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab [term 2 LEADER]: Becoming Leader. State: Replica: 02b1526aa9704241b56acb1aa58388ab, State: Running, Role: LEADER
I20260924 07:12:00.503645 13173 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:12:00.503666 13177 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b1526aa9704241b56acb1aa58388ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38983 } }
I20260924 07:12:00.503916 13177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02b1526aa9704241b56acb1aa58388ab. Latest consensus state: current_term: 2 leader_uuid: "02b1526aa9704241b56acb1aa58388ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b1526aa9704241b56acb1aa58388ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38983 } } }
I20260924 07:12:00.503980 13177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab [sys.catalog]: This master's current role is: LEADER
I20260924 07:12:00.504117 13178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "02b1526aa9704241b56acb1aa58388ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02b1526aa9704241b56acb1aa58388ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38983 } } }
I20260924 07:12:00.504191 13178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab [sys.catalog]: This master's current role is: LEADER
I20260924 07:12:00.504401 13182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:12:00.504911 13182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:12:00.505072 13182 catalog_manager.cc:1259] Loaded cluster ID: 9dd2cb1e9e6b4879a2d138d5e4ce2160
I20260924 07:12:00.505149 13182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:12:00.505208 13173 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 07:12:00.508884 13182 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:12:00.509429 13182 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab: Loaded TSK: 0
I20260924 07:12:00.509714 13182 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 07:12:00.598704 13175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:00.598973 13175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:00.599011 13175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:00.600632 13175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:00.600682 13175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:00.600719 13175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:12:00.602803 13175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56931/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35947
--tserver_master_addrs=127.0.98.62:38983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:12:00.603032 13175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:00.603274 13175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:00.610021 13211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:00.610132 13213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:00.610039 13210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:00.610725 13175 server_base.cc:1034] running on GCE node
I20260924 07:12:00.610947 13175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:00.611245 13175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:00.614535 13175 hybrid_clock.cc:648] HybridClock initialized: now 1790233920613654 us; error 894 us; skew 500 ppm
Sep 24 07:12:00 dist-test-slave-14ww krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:12:00.617820 13175 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:12:00.618341 13175 webserver.cc:460] Webserver started at http://127.0.98.1:35947/ using document root <none> and password file <none>
I20260924 07:12:00.618607 13175 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:00.618688 13175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 07:12:00 dist-test-slave-14ww krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 07:12:00.643932 13175 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:12:00.644876 13221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:00.645084 13175 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 07:12:00.645165 13175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-0/wal
uuid: "0b80daed62c048bf932ad2cda642ac03"
format_stamp: "Formatted at 2026-09-24 07:11:59 on dist-test-slave-14ww"
server_key: "047dd54772f1128627bc9204c577c6de"
server_key_iv: "76dcba85dd9609b49fab256ce277fc4c"
server_key_version: "kuduclusterkey@0"
I20260924 07:12:00.645603 13175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:00.666188 13175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:00.668366 13175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:00.668519 13175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:00.668709 13175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:12:00.669061 13175 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:12:00.669164 13175 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:00.669193 13175 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:12:00.669247 13175 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:00.675678 13175 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45281
I20260924 07:12:00.676164 13175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:12:00.679080 13334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45281 every 8 connection(s)
I20260924 07:12:00.679383 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 13175
I20260924 07:12:00.679663 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56931/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43167
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38983
--webserver_port=38313
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56931/kms with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:12:00 dist-test-slave-14ww krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:12:00.698185 13339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:00.682223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59215 (local address 127.0.98.62:38983)
0924 07:12:00.685618 (+ 3395us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:00.685627 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:00.685675 (+ 48us) server_negotiation.cc:407] Connection header received
0924 07:12:00.685824 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:00.685831 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:00.685888 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:00.686049 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:00.686725 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:00.688991 (+ 2266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:00.690652 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:00.691345 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:00.691464 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:00.694167 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:00.694171 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:00.694172 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:00.694184 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:00.696245 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:00.696720 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:00.696724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:00.696724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:00.696854 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:00.697171 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:00.697172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:00.697173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:00.697468 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:00.697564 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:00.697882 (+ 318us) server_negotiation.cc:299] Negotiation successful
0924 07:12:00.697994 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3337,"thread_start_us":73,"threads_started":1}
I20260924 07:12:00.698189 13337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:00.677457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38983 (local address 127.0.98.1:59215)
0924 07:12:00.680818 (+ 3361us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:00.680873 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:00.681613 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:00.686113 (+ 4500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:00.686117 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:00.686348 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:00.686613 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:00.686618 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:00.689123 (+ 2505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:00.689124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:00.690547 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:00.690550 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:00.691468 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:00.691470 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:00.691608 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:00.692256 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:00.692265 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:00.694028 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:12:00.696333 (+ 2305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:00.696339 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:00.696342 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:00.696620 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:00.696956 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:00.696956 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:00.696957 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:00.697090 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:00.697558 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:12:00.697564 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:12:00.697712 (+ 148us) client_negotiation.cc:769] Sending connection context
0924 07:12:00.697818 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 07:12:00.697971 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3308,"thread_start_us":593,"threads_started":1}
I20260924 07:12:00.699251 13335 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38983
I20260924 07:12:00.699335 13335 heartbeater.cc:464] Registering TS with master...
I20260924 07:12:00.699504 13335 heartbeater.cc:511] Master 127.0.98.62:38983 requested a full tablet report, sending...
W20260924 07:12:00.700337 13335 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38983 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 07:12:00.700402 13335 heartbeater.cc:464] Registering TS with master...
I20260924 07:12:00.700461 13335 heartbeater.cc:511] Master 127.0.98.62:38983 requested a full tablet report, sending...
I20260924 07:12:00.700911 13335 heartbeater.cc:464] Registering TS with master...
I20260924 07:12:00.700997 13335 heartbeater.cc:511] Master 127.0.98.62:38983 requested a full tablet report, sending...
W20260924 07:12:00.701514 13335 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 07:12:00.788640 13338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:00.789012 13338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:00.789111 13338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:00.790279 13338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:00.790354 13338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:00.790479 13338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:12:00.792024 13338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56931/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43167
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38313
--tserver_master_addrs=127.0.98.62:38983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:12:00.792404 13338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:00.792686 13338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:00.798713 13347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:00.799463 13345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:00.799533 13344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:00.800544 13338 server_base.cc:1034] running on GCE node
I20260924 07:12:00.800781 13338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:00.801019 13338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:00.808461 13338 hybrid_clock.cc:648] HybridClock initialized: now 1790233920808435 us; error 37 us; skew 500 ppm
Sep 24 07:12:00 dist-test-slave-14ww krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:12:00.811185 13338 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:12:00.811746 13338 webserver.cc:460] Webserver started at http://127.0.98.2:38313/ using document root <none> and password file <none>
I20260924 07:12:00.811993 13338 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:00.812073 13338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 07:12:00 dist-test-slave-14ww krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 07:12:00.826984 13338 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:12:00.827688 13355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:00.828673 13338 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:12:00.828791 13338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-1/wal
uuid: "f41e19b9881f4e06bed642d711b719b7"
format_stamp: "Formatted at 2026-09-24 07:11:59 on dist-test-slave-14ww"
server_key: "3a119271f5ecea765a877db881536ab0"
server_key_iv: "14d27984c69d4bbb3e621031d58b2a97"
server_key_version: "kuduclusterkey@0"
I20260924 07:12:00.829247 13338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:00.849659 13338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:00.851845 13338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:00.852005 13338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:00.852294 13338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:12:00.852675 13338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:12:00.852756 13338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:00.852792 13338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:12:00.852854 13338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:00.860268 13338 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43167
I20260924 07:12:00.860775 13338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:12:00.860873 13468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43167 every 8 connection(s)
I20260924 07:12:00.863549 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 13338
I20260924 07:12:00.863842 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56931/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37909
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38983
--webserver_port=44529
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56931/kms with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 07:12:00 dist-test-slave-14ww krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:12:00.872107 13339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:00.862064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35281 (local address 127.0.98.62:38983)
0924 07:12:00.862111 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:00.862113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:00.862125 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:12:00.862842 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:00.862843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:00.862856 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:00.862918 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:00.863649 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:00.864454 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:00.865937 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:00.866210 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:00.866296 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:00.868756 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:00.868758 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:00.868759 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:00.868767 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:00.870532 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:00.871014 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:00.871015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:00.871016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:00.871075 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:00.871350 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:00.871351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:00.871352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:00.871489 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:00.871534 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:00.871915 (+ 381us) server_negotiation.cc:299] Negotiation successful
0924 07:12:00.872011 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 07:12:00.872586 13471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:00.861784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38983 (local address 127.0.98.2:35281)
0924 07:12:00.861965 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:00.862043 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:00.862664 (+ 621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:00.863009 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:00.863013 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:00.863248 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:00.863535 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:00.863541 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:00.864584 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:00.864585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:00.865856 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:00.865859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:00.866318 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:00.866320 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:00.866465 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:00.867043 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:00.867053 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:00.868609 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:12:00.870624 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:00.870630 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:00.870633 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:00.870911 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:00.871147 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:00.871147 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:00.871148 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:00.871273 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:00.871584 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:12:00.871590 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:12:00.871759 (+ 169us) client_negotiation.cc:769] Sending connection context
0924 07:12:00.872203 (+ 444us) client_negotiation.cc:240] Negotiation successful
0924 07:12:00.872357 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
I20260924 07:12:00.873080 13469 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38983
I20260924 07:12:00.873175 13469 heartbeater.cc:464] Registering TS with master...
I20260924 07:12:00.873351 13469 heartbeater.cc:511] Master 127.0.98.62:38983 requested a full tablet report, sending...
W20260924 07:12:00.873951 13469 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38983 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 07:12:00.874035 13469 heartbeater.cc:464] Registering TS with master...
I20260924 07:12:00.874107 13469 heartbeater.cc:511] Master 127.0.98.62:38983 requested a full tablet report, sending...
I20260924 07:12:00.874388 13469 heartbeater.cc:464] Registering TS with master...
I20260924 07:12:00.874461 13469 heartbeater.cc:511] Master 127.0.98.62:38983 requested a full tablet report, sending...
W20260924 07:12:00.874732 13469 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 07:12:00.985357 13472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:00.985662 13472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:00.985702 13472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:00.987454 13472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:00.987512 13472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:00.987552 13472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:12:00.989775 13472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56931/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37909
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44529
--tserver_master_addrs=127.0.98.62:38983
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:12:00.990018 13472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:00.990289 13472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:00.996600 13477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:00.996608 13478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:00.996928 13480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:00.997766 13472 server_base.cc:1034] running on GCE node
I20260924 07:12:00.998055 13472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:00.998296 13472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:00.999454 13472 hybrid_clock.cc:648] HybridClock initialized: now 1790233920999427 us; error 46 us; skew 500 ppm
Sep 24 07:12:00 dist-test-slave-14ww krb5kdc[12141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:12:01.002251 13472 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:12:01.002750 13472 webserver.cc:460] Webserver started at http://127.0.98.3:44529/ using document root <none> and password file <none>
I20260924 07:12:01.002991 13472 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:01.003057 13472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 07:12:01 dist-test-slave-14ww krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 07:12:01.020366 13472 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 07:12:01.021469 13489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:01.021690 13472 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:12:01.021795 13472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-2/wal
uuid: "0041dfc393244a119e6a7a1ce733b349"
format_stamp: "Formatted at 2026-09-24 07:12:00 on dist-test-slave-14ww"
server_key: "653b8d3f5e75b723eec09a43cf3b3d19"
server_key_iv: "4da447bd65268d1d6004b37d614eeb80"
server_key_version: "kuduclusterkey@0"
I20260924 07:12:01.022256 13472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:01.041325 13472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:01.043229 13472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:01.043382 13472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:01.043617 13472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:12:01.043972 13472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:12:01.044032 13472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:01.044068 13472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:12:01.044096 13472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:01.050398 13472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37909
I20260924 07:12:01.050797 13472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:12:01.051316 13602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37909 every 8 connection(s)
Sep 24 07:12:01 dist-test-slave-14ww krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:12:01.061064 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 13472
I20260924 07:12:01.062191 13339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:01.051903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45747 (local address 127.0.98.62:38983)
0924 07:12:01.051956 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:01.051958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:01.051969 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:12:01.052459 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:01.052461 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:01.052473 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:01.052548 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:01.053144 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:01.053900 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:01.055348 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:01.055658 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:01.055739 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:01.058738 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:01.058740 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:01.058740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:01.058748 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:01.060560 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:01.061078 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:01.061079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:01.061079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:01.061137 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:01.061425 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:01.061426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:01.061427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:01.061577 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:01.061630 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:01.061995 (+ 365us) server_negotiation.cc:299] Negotiation successful
0924 07:12:01.062092 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 07:12:01.062682 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:01.051602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38983 (local address 127.0.98.3:45747)
0924 07:12:01.051794 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:01.051843 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:01.052342 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:01.052597 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:01.052601 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:01.052847 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:01.053044 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:01.053049 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:01.053993 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:01.053994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:01.055240 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:01.055243 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:01.055776 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:01.055777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:01.055920 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:01.056484 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:01.056493 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:01.058623 (+ 2130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:12:01.060655 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:01.060662 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:01.060664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:01.060988 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:01.061210 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:01.061211 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:01.061212 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:01.061344 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:01.061662 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:12:01.061667 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:12:01.061820 (+ 153us) client_negotiation.cc:769] Sending connection context
0924 07:12:01.062302 (+ 482us) client_negotiation.cc:240] Negotiation successful
0924 07:12:01.062459 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":88,"threads_started":1}
I20260924 07:12:01.064824 13603 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38983
I20260924 07:12:01.064980 13603 heartbeater.cc:464] Registering TS with master...
I20260924 07:12:01.065169 13603 heartbeater.cc:511] Master 127.0.98.62:38983 requested a full tablet report, sending...
W20260924 07:12:01.065829 13603 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38983 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 07:12:01.065935 13603 heartbeater.cc:464] Registering TS with master...
I20260924 07:12:01.066006 13603 heartbeater.cc:511] Master 127.0.98.62:38983 requested a full tablet report, sending...
I20260924 07:12:01.066314 13603 heartbeater.cc:464] Registering TS with master...
I20260924 07:12:01.066390 13603 heartbeater.cc:511] Master 127.0.98.62:38983 requested a full tablet report, sending...
W20260924 07:12:01.066675 13603 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 07:12:01.074620 13339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:01.061460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57774 (local address 127.0.98.62:38983)
0924 07:12:01.062269 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:01.062272 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:01.065761 (+ 3489us) server_negotiation.cc:407] Connection header received
0924 07:12:01.065785 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:01.065786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:01.065795 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:01.065848 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:01.068476 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:01.069072 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:01.070172 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:01.070469 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:01.070544 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:01.072035 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:01.072036 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:01.072037 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:01.072044 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:01.073545 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:01.073954 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:01.073954 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:01.073955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:01.074003 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:01.074207 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:01.074208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:01.074208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:01.074297 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:01.074408 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:01.074505 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 07:12:01.074557 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":792,"thread_start_us":64,"threads_started":1}
Sep 24 07:12:01 dist-test-slave-14ww krb5kdc[12141](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790233921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:12:01.701694 13335 heartbeater.cc:464] Registering TS with master...
I20260924 07:12:01.701843 13335 heartbeater.cc:511] Master 127.0.98.62:38983 requested a full tablet report, sending...
I20260924 07:12:01.702225 13137 ts_manager.cc:194] Registered new tserver with Master: 0b80daed62c048bf932ad2cda642ac03 (127.0.98.1:45281)
I20260924 07:12:01.703253 13137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59215
I20260924 07:12:01.874931 13469 heartbeater.cc:464] Registering TS with master...
I20260924 07:12:01.875077 13469 heartbeater.cc:511] Master 127.0.98.62:38983 requested a full tablet report, sending...
I20260924 07:12:01.875355 13137 ts_manager.cc:194] Registered new tserver with Master: f41e19b9881f4e06bed642d711b719b7 (127.0.98.2:43167)
I20260924 07:12:01.875867 13137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35281
Sep 24 07:12:01 dist-test-slave-14ww krb5kdc[12141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 07:12:02.066856 13603 heartbeater.cc:464] Registering TS with master...
I20260924 07:12:02.067003 13603 heartbeater.cc:511] Master 127.0.98.62:38983 requested a full tablet report, sending...
I20260924 07:12:02.067286 13137 ts_manager.cc:194] Registered new tserver with Master: 0041dfc393244a119e6a7a1ce733b349 (127.0.98.3:37909)
I20260924 07:12:02.067797 13137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45747
I20260924 07:12:02.074224 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:12:02.083130 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:02.075172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57784 (local address 127.0.98.62:38983)
0924 07:12:02.075441 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:02.075443 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:02.075450 (+ 7us) server_negotiation.cc:407] Connection header received
0924 07:12:02.075473 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:02.075473 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:02.075485 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:02.075559 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:02.076290 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:02.077133 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:02.078188 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:02.078508 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:02.078588 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:02.080001 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:02.080003 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:02.080003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:02.080011 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:02.081842 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:02.082213 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:02.082215 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:02.082215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:02.082280 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:02.082480 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:02.082481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:02.082481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:02.082620 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:02.082728 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:02.082849 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 07:12:02.082934 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":62,"threads_started":1}
I20260924 07:12:02.085390 13137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:12:02.087527 13137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:12:02.095202 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:02.090439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43167 (local address 127.0.0.1:33280)
0924 07:12:02.090581 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:02.090620 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:02.090688 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:02.091219 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:02.091222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:02.091224 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:12:02.091283 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:02.091285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:02.092641 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:02.092642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:02.093944 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:02.093947 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:02.094919 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:02.094921 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:02.095004 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:02.095019 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 07:12:02.095070 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 07:12:02.095118 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":80,"threads_started":1}
I20260924 07:12:02.095924 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:02.090888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33280 (local address 127.0.98.2:43167)
0924 07:12:02.091032 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:02.091037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:02.091048 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:12:02.091084 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:02.091086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:02.091102 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:02.091374 (+ 272us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:12:02.091444 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:02.092539 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:02.094047 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:02.094793 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:02.095693 (+ 900us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:02.095779 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:02.095805 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 07:12:02.095844 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260924 07:12:02.096138 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:02.090754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47930 (local address 127.0.98.1:45281)
0924 07:12:02.090865 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:02.090870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:02.091326 (+ 456us) server_negotiation.cc:407] Connection header received
0924 07:12:02.091430 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:02.091432 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:02.091447 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:02.091512 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:12:02.091710 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:02.093149 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:02.094564 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:02.095659 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:02.095721 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:02.095799 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:02.096014 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 07:12:02.096059 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":59,"threads_started":1}
I20260924 07:12:02.096812 13403 tablet_service.cc:1467] Processing CreateTablet for tablet 69d61f599f0949c0ac006afbab08d3f4 (DEFAULT_TABLE table=test-table [id=43e71b0e51c145029f0ce83fa348c998]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:12:02.097172 13403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69d61f599f0949c0ac006afbab08d3f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:12:02.100462 13636 tablet_bootstrap.cc:492] T 69d61f599f0949c0ac006afbab08d3f4 P f41e19b9881f4e06bed642d711b719b7: Bootstrap starting.
I20260924 07:12:02.101056 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:02.090647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45281 (local address 127.0.0.1:47930)
0924 07:12:02.091266 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:02.091275 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:02.091353 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:02.091600 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:02.091600 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:02.091602 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:12:02.091638 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:02.091639 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:02.093286 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:02.093287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:02.094448 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:02.094451 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:02.095862 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:02.095863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:02.095928 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:02.095932 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:12:02.100921 (+ 4989us) client_negotiation.cc:240] Negotiation successful
0924 07:12:02.100978 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":52,"threads_started":1}
I20260924 07:12:02.101678 13636 tablet_bootstrap.cc:654] T 69d61f599f0949c0ac006afbab08d3f4 P f41e19b9881f4e06bed642d711b719b7: Neither blocks nor log segments found. Creating new log.
I20260924 07:12:02.101867 13264 tablet_service.cc:1467] Processing CreateTablet for tablet 69d61f599f0949c0ac006afbab08d3f4 (DEFAULT_TABLE table=test-table [id=43e71b0e51c145029f0ce83fa348c998]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:12:02.102018 13636 log.cc:824] T 69d61f599f0949c0ac006afbab08d3f4 P f41e19b9881f4e06bed642d711b719b7: Log is configured to *not* fsync() on all Append() calls
I20260924 07:12:02.102154 13264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69d61f599f0949c0ac006afbab08d3f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:12:02.104490 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:02.090768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37909 (local address 127.0.0.1:56004)
0924 07:12:02.095272 (+ 4504us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:02.095283 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:02.095322 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:02.100996 (+ 5674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:02.100997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:02.100999 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:12:02.101026 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:02.101028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:02.102289 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:02.102290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:02.103367 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:02.103370 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:02.104286 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:02.104287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:02.104354 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:02.104357 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:12:02.104390 (+ 33us) client_negotiation.cc:240] Negotiation successful
0924 07:12:02.104427 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4494,"thread_start_us":55,"threads_started":1}
I20260924 07:12:02.104825 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:02.090870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56004 (local address 127.0.98.3:37909)
0924 07:12:02.100816 (+ 9946us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:02.100821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:02.100831 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:12:02.100866 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:02.100868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:02.100884 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:02.101101 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:12:02.101194 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:02.102197 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:02.103443 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:02.104180 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:02.104594 (+ 414us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:02.104659 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:02.104685 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 07:12:02.104724 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9907,"thread_start_us":52,"threads_started":1}
I20260924 07:12:02.105427 13638 tablet_bootstrap.cc:492] T 69d61f599f0949c0ac006afbab08d3f4 P 0b80daed62c048bf932ad2cda642ac03: Bootstrap starting.
I20260924 07:12:02.105383 13537 tablet_service.cc:1467] Processing CreateTablet for tablet 69d61f599f0949c0ac006afbab08d3f4 (DEFAULT_TABLE table=test-table [id=43e71b0e51c145029f0ce83fa348c998]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:12:02.105687 13537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69d61f599f0949c0ac006afbab08d3f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:12:02.106017 13638 tablet_bootstrap.cc:654] T 69d61f599f0949c0ac006afbab08d3f4 P 0b80daed62c048bf932ad2cda642ac03: Neither blocks nor log segments found. Creating new log.
I20260924 07:12:02.106323 13638 log.cc:824] T 69d61f599f0949c0ac006afbab08d3f4 P 0b80daed62c048bf932ad2cda642ac03: Log is configured to *not* fsync() on all Append() calls
I20260924 07:12:02.107110 13638 tablet_bootstrap.cc:492] T 69d61f599f0949c0ac006afbab08d3f4 P 0b80daed62c048bf932ad2cda642ac03: No bootstrap required, opened a new log
I20260924 07:12:02.107184 13638 ts_tablet_manager.cc:1397] T 69d61f599f0949c0ac006afbab08d3f4 P 0b80daed62c048bf932ad2cda642ac03: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:12:02.109266 13640 tablet_bootstrap.cc:492] T 69d61f599f0949c0ac006afbab08d3f4 P 0041dfc393244a119e6a7a1ce733b349: Bootstrap starting.
I20260924 07:12:02.109920 13640 tablet_bootstrap.cc:654] T 69d61f599f0949c0ac006afbab08d3f4 P 0041dfc393244a119e6a7a1ce733b349: Neither blocks nor log segments found. Creating new log.
I20260924 07:12:02.109922 13638 raft_consensus.cc:348] T 69d61f599f0949c0ac006afbab08d3f4 P 0b80daed62c048bf932ad2cda642ac03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41e19b9881f4e06bed642d711b719b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43167 } } peers { permanent_uuid: "0b80daed62c048bf932ad2cda642ac03" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45281 } } peers { permanent_uuid: "0041dfc393244a119e6a7a1ce733b349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37909 } }
I20260924 07:12:02.110158 13638 raft_consensus.cc:374] T 69d61f599f0949c0ac006afbab08d3f4 P 0b80daed62c048bf932ad2cda642ac03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:12:02.110241 13638 raft_consensus.cc:729] T 69d61f599f0949c0ac006afbab08d3f4 P 0b80daed62c048bf932ad2cda642ac03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b80daed62c048bf932ad2cda642ac03, State: Initialized, Role: FOLLOWER
I20260924 07:12:02.110275 13640 log.cc:824] T 69d61f599f0949c0ac006afbab08d3f4 P 0041dfc393244a119e6a7a1ce733b349: Log is configured to *not* fsync() on all Append() calls
I20260924 07:12:02.110451 13638 consensus_queue.cc:260] T 69d61f599f0949c0ac006afbab08d3f4 P 0b80daed62c048bf932ad2cda642ac03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41e19b9881f4e06bed642d711b719b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43167 } } peers { permanent_uuid: "0b80daed62c048bf932ad2cda642ac03" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45281 } } peers { permanent_uuid: "0041dfc393244a119e6a7a1ce733b349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37909 } }
I20260924 07:12:02.110798 13638 ts_tablet_manager.cc:1428] T 69d61f599f0949c0ac006afbab08d3f4 P 0b80daed62c048bf932ad2cda642ac03: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260924 07:12:02.110956 13335 heartbeater.cc:503] Master 127.0.98.62:38983 was elected leader, sending a full tablet report...
I20260924 07:12:02.111023 13640 tablet_bootstrap.cc:492] T 69d61f599f0949c0ac006afbab08d3f4 P 0041dfc393244a119e6a7a1ce733b349: No bootstrap required, opened a new log
I20260924 07:12:02.111073 13640 ts_tablet_manager.cc:1397] T 69d61f599f0949c0ac006afbab08d3f4 P 0041dfc393244a119e6a7a1ce733b349: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:12:02.113756 13636 tablet_bootstrap.cc:492] T 69d61f599f0949c0ac006afbab08d3f4 P f41e19b9881f4e06bed642d711b719b7: No bootstrap required, opened a new log
I20260924 07:12:02.113688 13640 raft_consensus.cc:348] T 69d61f599f0949c0ac006afbab08d3f4 P 0041dfc393244a119e6a7a1ce733b349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41e19b9881f4e06bed642d711b719b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43167 } } peers { permanent_uuid: "0b80daed62c048bf932ad2cda642ac03" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45281 } } peers { permanent_uuid: "0041dfc393244a119e6a7a1ce733b349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37909 } }
I20260924 07:12:02.113824 13636 ts_tablet_manager.cc:1397] T 69d61f599f0949c0ac006afbab08d3f4 P f41e19b9881f4e06bed642d711b719b7: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260924 07:12:02.113826 13640 raft_consensus.cc:374] T 69d61f599f0949c0ac006afbab08d3f4 P 0041dfc393244a119e6a7a1ce733b349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:12:02.113874 13640 raft_consensus.cc:729] T 69d61f599f0949c0ac006afbab08d3f4 P 0041dfc393244a119e6a7a1ce733b349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0041dfc393244a119e6a7a1ce733b349, State: Initialized, Role: FOLLOWER
I20260924 07:12:02.114070 13640 consensus_queue.cc:260] T 69d61f599f0949c0ac006afbab08d3f4 P 0041dfc393244a119e6a7a1ce733b349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41e19b9881f4e06bed642d711b719b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43167 } } peers { permanent_uuid: "0b80daed62c048bf932ad2cda642ac03" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45281 } } peers { permanent_uuid: "0041dfc393244a119e6a7a1ce733b349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37909 } }
I20260924 07:12:02.114346 13640 ts_tablet_manager.cc:1428] T 69d61f599f0949c0ac006afbab08d3f4 P 0041dfc393244a119e6a7a1ce733b349: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 07:12:02.114463 13603 heartbeater.cc:503] Master 127.0.98.62:38983 was elected leader, sending a full tablet report...
I20260924 07:12:02.116164 13636 raft_consensus.cc:348] T 69d61f599f0949c0ac006afbab08d3f4 P f41e19b9881f4e06bed642d711b719b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41e19b9881f4e06bed642d711b719b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43167 } } peers { permanent_uuid: "0b80daed62c048bf932ad2cda642ac03" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45281 } } peers { permanent_uuid: "0041dfc393244a119e6a7a1ce733b349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37909 } }
I20260924 07:12:02.116271 13636 raft_consensus.cc:374] T 69d61f599f0949c0ac006afbab08d3f4 P f41e19b9881f4e06bed642d711b719b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:12:02.116289 13636 raft_consensus.cc:729] T 69d61f599f0949c0ac006afbab08d3f4 P f41e19b9881f4e06bed642d711b719b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f41e19b9881f4e06bed642d711b719b7, State: Initialized, Role: FOLLOWER
I20260924 07:12:02.116410 13636 consensus_queue.cc:260] T 69d61f599f0949c0ac006afbab08d3f4 P f41e19b9881f4e06bed642d711b719b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41e19b9881f4e06bed642d711b719b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43167 } } peers { permanent_uuid: "0b80daed62c048bf932ad2cda642ac03" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45281 } } peers { permanent_uuid: "0041dfc393244a119e6a7a1ce733b349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37909 } }
I20260924 07:12:02.116654 13636 ts_tablet_manager.cc:1428] T 69d61f599f0949c0ac006afbab08d3f4 P f41e19b9881f4e06bed642d711b719b7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 07:12:02.116667 13469 heartbeater.cc:503] Master 127.0.98.62:38983 was elected leader, sending a full tablet report...
I20260924 07:12:02.126256 13643 raft_consensus.cc:482] T 69d61f599f0949c0ac006afbab08d3f4 P 0041dfc393244a119e6a7a1ce733b349 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:12:02.126353 13643 raft_consensus.cc:504] T 69d61f599f0949c0ac006afbab08d3f4 P 0041dfc393244a119e6a7a1ce733b349 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41e19b9881f4e06bed642d711b719b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43167 } } peers { permanent_uuid: "0b80daed62c048bf932ad2cda642ac03" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45281 } } peers { permanent_uuid: "0041dfc393244a119e6a7a1ce733b349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37909 } }
I20260924 07:12:02.126698 13643 leader_election.cc:288] T 69d61f599f0949c0ac006afbab08d3f4 P 0041dfc393244a119e6a7a1ce733b349 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f41e19b9881f4e06bed642d711b719b7 (127.0.98.2:43167), 0b80daed62c048bf932ad2cda642ac03 (127.0.98.1:45281)
I20260924 07:12:02.130085 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:02.127171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52907 (local address 127.0.98.2:43167)
0924 07:12:02.127254 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:02.127256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:02.127263 (+ 7us) server_negotiation.cc:407] Connection header received
0924 07:12:02.127283 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:02.127284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:02.127293 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:02.127366 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:12:02.127537 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:02.128033 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:02.129091 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:02.129673 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:02.129720 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:02.129733 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:02.129973 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 07:12:02.130013 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 07:12:02.130574 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:02.126829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43167 (local address 127.0.98.3:52907)
0924 07:12:02.126976 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:02.126987 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:02.127035 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:02.127369 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:02.127370 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:02.127372 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:12:02.127449 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:02.127452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:02.128109 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:02.128110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:02.129011 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:02.129013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:02.129806 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:02.129807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:02.129892 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:02.129896 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:12:02.130435 (+ 539us) client_negotiation.cc:240] Negotiation successful
0924 07:12:02.130476 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":4,"thread_start_us":100,"threads_started":1}
I20260924 07:12:02.130959 13423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d61f599f0949c0ac006afbab08d3f4" candidate_uuid: "0041dfc393244a119e6a7a1ce733b349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f41e19b9881f4e06bed642d711b719b7" is_pre_election: true
I20260924 07:12:02.131096 13423 raft_consensus.cc:2454] T 69d61f599f0949c0ac006afbab08d3f4 P f41e19b9881f4e06bed642d711b719b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0041dfc393244a119e6a7a1ce733b349 in term 0.
I20260924 07:12:02.131369 13490 leader_election.cc:302] T 69d61f599f0949c0ac006afbab08d3f4 P 0041dfc393244a119e6a7a1ce733b349 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0041dfc393244a119e6a7a1ce733b349, f41e19b9881f4e06bed642d711b719b7; no voters:
I20260924 07:12:02.131480 13643 raft_consensus.cc:2781] T 69d61f599f0949c0ac006afbab08d3f4 P 0041dfc393244a119e6a7a1ce733b349 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:12:02.131517 13643 raft_consensus.cc:482] T 69d61f599f0949c0ac006afbab08d3f4 P 0041dfc393244a119e6a7a1ce733b349 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:12:02.131538 13643 raft_consensus.cc:3037] T 69d61f599f0949c0ac006afbab08d3f4 P 0041dfc393244a119e6a7a1ce733b349 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:12:02.132280 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:02.127430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36255 (local address 127.0.98.1:45281)
0924 07:12:02.127469 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:02.127471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:02.127478 (+ 7us) server_negotiation.cc:407] Connection header received
0924 07:12:02.127501 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:02.127502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:02.127512 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:02.127556 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:12:02.128979 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:02.129693 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:02.131330 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:02.131893 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:02.131939 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:02.131948 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:02.132168 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 07:12:02.132214 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 07:12:02.132448 13643 raft_consensus.cc:504] T 69d61f599f0949c0ac006afbab08d3f4 P 0041dfc393244a119e6a7a1ce733b349 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41e19b9881f4e06bed642d711b719b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43167 } } peers { permanent_uuid: "0b80daed62c048bf932ad2cda642ac03" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45281 } } peers { permanent_uuid: "0041dfc393244a119e6a7a1ce733b349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37909 } }
I20260924 07:12:02.132532 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:02.127032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45281 (local address 127.0.98.3:36255)
0924 07:12:02.127154 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:02.127163 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:02.127204 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:02.128827 (+ 1623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:02.128828 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:02.128830 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:12:02.128889 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:02.128890 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:02.129783 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:02.129784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:02.130934 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:02.130937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:02.132004 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:02.132005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:02.132085 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:02.132088 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:12:02.132377 (+ 289us) client_negotiation.cc:240] Negotiation successful
0924 07:12:02.132422 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":22,"thread_start_us":88,"threads_started":1}
I20260924 07:12:02.132575 13643 leader_election.cc:288] T 69d61f599f0949c0ac006afbab08d3f4 P 0041dfc393244a119e6a7a1ce733b349 [CANDIDATE]: Term 1 election: Requested vote from peers f41e19b9881f4e06bed642d711b719b7 (127.0.98.2:43167), 0b80daed62c048bf932ad2cda642ac03 (127.0.98.1:45281)
I20260924 07:12:02.132901 13283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d61f599f0949c0ac006afbab08d3f4" candidate_uuid: "0041dfc393244a119e6a7a1ce733b349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b80daed62c048bf932ad2cda642ac03"
I20260924 07:12:02.133008 13283 raft_consensus.cc:3037] T 69d61f599f0949c0ac006afbab08d3f4 P 0b80daed62c048bf932ad2cda642ac03 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:12:02.133867 13283 raft_consensus.cc:2454] T 69d61f599f0949c0ac006afbab08d3f4 P 0b80daed62c048bf932ad2cda642ac03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0041dfc393244a119e6a7a1ce733b349 in term 1.
I20260924 07:12:02.134049 13273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d61f599f0949c0ac006afbab08d3f4" candidate_uuid: "0041dfc393244a119e6a7a1ce733b349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b80daed62c048bf932ad2cda642ac03" is_pre_election: true
I20260924 07:12:02.134085 13490 leader_election.cc:302] T 69d61f599f0949c0ac006afbab08d3f4 P 0041dfc393244a119e6a7a1ce733b349 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0041dfc393244a119e6a7a1ce733b349, 0b80daed62c048bf932ad2cda642ac03; no voters:
I20260924 07:12:02.134132 13273 raft_consensus.cc:2362] T 69d61f599f0949c0ac006afbab08d3f4 P 0b80daed62c048bf932ad2cda642ac03 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0041dfc393244a119e6a7a1ce733b349 in term 1. Re-sending same reply.
I20260924 07:12:02.134155 13643 raft_consensus.cc:2781] T 69d61f599f0949c0ac006afbab08d3f4 P 0041dfc393244a119e6a7a1ce733b349 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:12:02.134284 13643 raft_consensus.cc:686] T 69d61f599f0949c0ac006afbab08d3f4 P 0041dfc393244a119e6a7a1ce733b349 [term 1 LEADER]: Becoming Leader. State: Replica: 0041dfc393244a119e6a7a1ce733b349, State: Running, Role: LEADER
I20260924 07:12:02.134368 13643 consensus_queue.cc:237] T 69d61f599f0949c0ac006afbab08d3f4 P 0041dfc393244a119e6a7a1ce733b349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41e19b9881f4e06bed642d711b719b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43167 } } peers { permanent_uuid: "0b80daed62c048bf932ad2cda642ac03" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45281 } } peers { permanent_uuid: "0041dfc393244a119e6a7a1ce733b349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37909 } }
I20260924 07:12:02.135183 13137 catalog_manager.cc:5495] T 69d61f599f0949c0ac006afbab08d3f4 P 0041dfc393244a119e6a7a1ce733b349 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0041dfc393244a119e6a7a1ce733b349 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0041dfc393244a119e6a7a1ce733b349" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f41e19b9881f4e06bed642d711b719b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b80daed62c048bf932ad2cda642ac03" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0041dfc393244a119e6a7a1ce733b349" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37909 } health_report { overall_health: HEALTHY } } }
I20260924 07:12:02.135592 13423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69d61f599f0949c0ac006afbab08d3f4" candidate_uuid: "0041dfc393244a119e6a7a1ce733b349" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f41e19b9881f4e06bed642d711b719b7"
I20260924 07:12:02.135668 13423 raft_consensus.cc:3037] T 69d61f599f0949c0ac006afbab08d3f4 P f41e19b9881f4e06bed642d711b719b7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:12:02.136363 13423 raft_consensus.cc:2454] T 69d61f599f0949c0ac006afbab08d3f4 P f41e19b9881f4e06bed642d711b719b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0041dfc393244a119e6a7a1ce733b349 in term 1.
I20260924 07:12:02.144877 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:02.142419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56018 (local address 127.0.98.3:37909)
0924 07:12:02.142470 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:02.142472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:02.142484 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:12:02.142564 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:02.142565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:02.142582 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:02.142670 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:12:02.142771 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:02.143269 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:02.144060 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:02.144270 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:02.144355 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:02.144525 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:12:02.144627 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:12:02.144720 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:02.144783 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 07:12:02.144802 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 07:12:02.146426 13423 raft_consensus.cc:1261] T 69d61f599f0949c0ac006afbab08d3f4 P f41e19b9881f4e06bed642d711b719b7 [term 1 FOLLOWER]: Refusing update from remote peer 0041dfc393244a119e6a7a1ce733b349: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:12:02.146426 13273 raft_consensus.cc:1261] T 69d61f599f0949c0ac006afbab08d3f4 P 0b80daed62c048bf932ad2cda642ac03 [term 1 FOLLOWER]: Refusing update from remote peer 0041dfc393244a119e6a7a1ce733b349: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:12:02.148262 13643 consensus_queue.cc:1035] T 69d61f599f0949c0ac006afbab08d3f4 P 0041dfc393244a119e6a7a1ce733b349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b80daed62c048bf932ad2cda642ac03" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:12:02.148437 13643 consensus_queue.cc:1035] T 69d61f599f0949c0ac006afbab08d3f4 P 0041dfc393244a119e6a7a1ce733b349 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f41e19b9881f4e06bed642d711b719b7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:12:02.154134 13650 mvcc.cc:205] Tried to move back new op lower bound from 7332798145109512192 to 7332798145062948864. Current Snapshot: MvccSnapshot[applied={T|T < 7332798145109512192}]
I20260924 07:12:02.154275 13651 mvcc.cc:205] Tried to move back new op lower bound from 7332798145109512192 to 7332798145062948864. Current Snapshot: MvccSnapshot[applied={T|T < 7332798145109512192}]
I20260924 07:12:02.158039 13137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:12:02.158142 13137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:12:02.158936 13137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 02b1526aa9704241b56acb1aa58388ab: Sending DeleteTablet for 3 replicas of tablet 69d61f599f0949c0ac006afbab08d3f4
I20260924 07:12:02.159356 13403 tablet_service.cc:1514] Processing DeleteTablet for tablet 69d61f599f0949c0ac006afbab08d3f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:12:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33280
I20260924 07:12:02.159567 13264 tablet_service.cc:1514] Processing DeleteTablet for tablet 69d61f599f0949c0ac006afbab08d3f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:12:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47930
I20260924 07:12:02.159967 13537 tablet_service.cc:1514] Processing DeleteTablet for tablet 69d61f599f0949c0ac006afbab08d3f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:12:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56004
I20260924 07:12:02.160228 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 13175
I20260924 07:12:02.163450 13662 tablet_replica.cc:330] T 69d61f599f0949c0ac006afbab08d3f4 P 0041dfc393244a119e6a7a1ce733b349: stopping tablet replica
I20260924 07:12:02.163585 13662 raft_consensus.cc:2229] T 69d61f599f0949c0ac006afbab08d3f4 P 0041dfc393244a119e6a7a1ce733b349 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:12:02.163766 13662 raft_consensus.cc:2258] T 69d61f599f0949c0ac006afbab08d3f4 P 0041dfc393244a119e6a7a1ce733b349 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:12:02.164266 13662 ts_tablet_manager.cc:1905] T 69d61f599f0949c0ac006afbab08d3f4 P 0041dfc393244a119e6a7a1ce733b349: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:12:02.165393 13660 tablet_replica.cc:330] T 69d61f599f0949c0ac006afbab08d3f4 P f41e19b9881f4e06bed642d711b719b7: stopping tablet replica
I20260924 07:12:02.165532 13660 raft_consensus.cc:2229] T 69d61f599f0949c0ac006afbab08d3f4 P f41e19b9881f4e06bed642d711b719b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:12:02.165696 13660 raft_consensus.cc:2258] T 69d61f599f0949c0ac006afbab08d3f4 P f41e19b9881f4e06bed642d711b719b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:12:02.166118 13662 ts_tablet_manager.cc:1918] T 69d61f599f0949c0ac006afbab08d3f4 P 0041dfc393244a119e6a7a1ce733b349: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:12:02.166193 13660 ts_tablet_manager.cc:1905] T 69d61f599f0949c0ac006afbab08d3f4 P f41e19b9881f4e06bed642d711b719b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:12:02.166230 13662 log.cc:1196] T 69d61f599f0949c0ac006afbab08d3f4 P 0041dfc393244a119e6a7a1ce733b349: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-2/wal/wals/69d61f599f0949c0ac006afbab08d3f4
I20260924 07:12:02.166584 13662 ts_tablet_manager.cc:1939] T 69d61f599f0949c0ac006afbab08d3f4 P 0041dfc393244a119e6a7a1ce733b349: Deleting consensus metadata
I20260924 07:12:02.167228 13121 catalog_manager.cc:4841] TS 0041dfc393244a119e6a7a1ce733b349 (127.0.98.3:37909): tablet 69d61f599f0949c0ac006afbab08d3f4 (table test-table [id=43e71b0e51c145029f0ce83fa348c998]) successfully deleted
I20260924 07:12:02.167819 13660 ts_tablet_manager.cc:1918] T 69d61f599f0949c0ac006afbab08d3f4 P f41e19b9881f4e06bed642d711b719b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:12:02.167893 13660 log.cc:1196] T 69d61f599f0949c0ac006afbab08d3f4 P f41e19b9881f4e06bed642d711b719b7: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790233772407728-392-0/minicluster-data/ts-1/wal/wals/69d61f599f0949c0ac006afbab08d3f4
I20260924 07:12:02.168151 13660 ts_tablet_manager.cc:1939] T 69d61f599f0949c0ac006afbab08d3f4 P f41e19b9881f4e06bed642d711b719b7: Deleting consensus metadata
I20260924 07:12:02.168517 13121 catalog_manager.cc:4841] TS f41e19b9881f4e06bed642d711b719b7 (127.0.98.2:43167): tablet 69d61f599f0949c0ac006afbab08d3f4 (table test-table [id=43e71b0e51c145029f0ce83fa348c998]) successfully deleted
W20260924 07:12:02.178304 13121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45281 (error 108)
I20260924 07:12:02.179044 13630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:02.178907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45281 (local address 127.0.0.1:47932)
0924 07:12:02.178953 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:02.179006 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260924 07:12:02.179262 13121 catalog_manager.cc:4584] TS 0b80daed62c048bf932ad2cda642ac03 (127.0.98.1:45281): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 69d61f599f0949c0ac006afbab08d3f4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45281: connect: Connection refused (error 111)
I20260924 07:12:02.182431 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 13338
I20260924 07:12:02.188691 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 13472
I20260924 07:12:02.195070 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 13104
I20260924 07:12:02.201386 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 07:12:02.545334 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 07:12:02.545449 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 07:12:02.889550 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 07:12:02.889 UTC [12171] LOG: received smart shutdown request
2026-09-24 07:12:02.893 UTC [12171] LOG: background worker "logical replication launcher" (PID 12180) exited with exit code 1
2026-09-24 07:12:02.893 UTC [12175] LOG: shutting down
2026-09-24 07:12:03.082 UTC [12171] LOG: database system is shut down
2026-09-24T07:12:03Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46449 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:12:03 dist-test-slave-14ww krb5kdc[13673](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:12:03 dist-test-slave-14ww krb5kdc[13673](info): set up 2 sockets
Sep 24 07:12:03 dist-test-slave-14ww krb5kdc[13673](info): commencing operation
krb5kdc: starting...
W20260924 07:12:04.252651 392 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:12:04 dist-test-slave-14ww krb5kdc[13673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:12:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:12:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:12:04.369728 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45593
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45593,127.0.98.61:32901,127.0.98.60:38559
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:12:04.449132 13686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:04.449323 13686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:04.449344 13686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:04.450385 13686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:12:04.450421 13686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:04.450435 13686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:04.450448 13686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:12:04.450459 13686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:12:04.451855 13686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45593,127.0.98.61:32901,127.0.98.60:38559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:12:04.452030 13686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:04.452183 13686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:04.456688 13694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:04.456802 13686 server_base.cc:1034] running on GCE node
W20260924 07:12:04.456713 13691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:04.456688 13692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:04.457243 13686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:04.457521 13686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:04.458758 13686 hybrid_clock.cc:648] HybridClock initialized: now 1790233924458744 us; error 44 us; skew 500 ppm
Sep 24 07:12:04 dist-test-slave-14ww krb5kdc[13673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:12:04.461230 13686 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:12:04.461628 13686 webserver.cc:460] Webserver started at http://127.0.98.62:44979/ using document root <none> and password file <none>
I20260924 07:12:04.461848 13686 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:04.461912 13686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:04.462054 13686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:12:04.463020 13686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "68aa88f30b5947379e2c90835a39ba78"
format_stamp: "Formatted at 2026-09-24 07:12:04 on dist-test-slave-14ww"
server_key: "c08b17fb2a6eec92acfbc87f232c7a1f"
server_key_iv: "89dfaee3a8102af1b85cc97ee975a074"
server_key_version: "encryptionkey@0"
I20260924 07:12:04.463387 13686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "68aa88f30b5947379e2c90835a39ba78"
format_stamp: "Formatted at 2026-09-24 07:12:04 on dist-test-slave-14ww"
server_key: "c08b17fb2a6eec92acfbc87f232c7a1f"
server_key_iv: "89dfaee3a8102af1b85cc97ee975a074"
server_key_version: "encryptionkey@0"
I20260924 07:12:04.464615 13686 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 07:12:04.465332 13701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:04.465492 13686 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:04.465606 13686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "68aa88f30b5947379e2c90835a39ba78"
format_stamp: "Formatted at 2026-09-24 07:12:04 on dist-test-slave-14ww"
server_key: "c08b17fb2a6eec92acfbc87f232c7a1f"
server_key_iv: "89dfaee3a8102af1b85cc97ee975a074"
server_key_version: "encryptionkey@0"
I20260924 07:12:04.465694 13686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:04.481815 13686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:04.483480 13686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:04.483624 13686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:04.487222 13686 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45593
I20260924 07:12:04.487253 13753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45593 every 8 connection(s)
I20260924 07:12:04.487582 13686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:12:04.488415 13754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:12:04.489938 13754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45593 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:12:04.493798 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 13686
I20260924 07:12:04.493907 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:12:04.494067 392 external_mini_cluster.cc:1428] Setting key eaa13dd10044c6b886d1e25509065035
Sep 24 07:12:04 dist-test-slave-14ww krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:12:04.499300 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.490549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45593 (local address 127.0.0.1:37156)
0924 07:12:04.490712 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:04.490746 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:04.491157 (+ 411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:04.491394 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:04.491396 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:04.491631 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:04.491777 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:04.491780 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.492710 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.492711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:04.493418 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:04.493420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.493871 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.493872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:04.493933 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:04.494441 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:04.494448 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:04.495896 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:12:04.497972 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:04.497978 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:04.497979 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:04.498159 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:04.498415 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:04.498415 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:04.498416 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:04.498487 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:04.498920 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:12:04.498924 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:12:04.498973 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 07:12:04.499041 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 07:12:04.499159 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":57,"threads_started":1}
I20260924 07:12:04.499419 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.490682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37156 (local address 127.0.98.62:45593)
0924 07:12:04.490800 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:04.490803 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:04.490828 (+ 25us) server_negotiation.cc:407] Connection header received
0924 07:12:04.491274 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:04.491276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:04.491291 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:04.491364 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:04.491853 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.492631 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.493485 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.493781 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.493907 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:04.495975 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:04.495983 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:04.495983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:04.495991 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:04.497891 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:04.498242 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:04.498245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:04.498245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:04.498346 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:04.498553 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:04.498554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:04.498555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:04.498847 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:04.498942 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:04.499096 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 07:12:04.499243 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":86,"thread_start_us":61,"threads_started":1}
I20260924 07:12:04.500103 13754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32901 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:12:04.500487 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.500363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32901 (local address 127.0.0.1:46086)
0924 07:12:04.500416 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:04.500468 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 07:12:04.500638 13703 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32901: connect: Connection refused (error 111)
I20260924 07:12:04.501056 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.500965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32901 (local address 127.0.0.1:46092)
0924 07:12:04.501010 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:04.501033 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 07:12:04.501155 13754 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:32901: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32901: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 07:12:04.520421 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:32901
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45593,127.0.98.61:32901,127.0.98.60:38559
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:12:04.550372 13754 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32901 } attempt: 1
I20260924 07:12:04.550843 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.550707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32901 (local address 127.0.0.1:46104)
0924 07:12:04.550776 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:04.550812 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 07:12:04.551179 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.551082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32901 (local address 127.0.0.1:46114)
0924 07:12:04.551131 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:04.551161 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 07:12:04.551258 13754 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:32901: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32901: connect: Connection refused (error 111)
W20260924 07:12:04.598140 13761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:04.598338 13761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:04.598361 13761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:04.599773 13761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:12:04.599823 13761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:04.599838 13761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:04.599851 13761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:12:04.599917 13761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:12:04.601552 13761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45593,127.0.98.61:32901,127.0.98.60:38559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:32901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:12:04.601816 13761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:04.602049 13761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:12:04.607059 13761 server_base.cc:1034] running on GCE node
W20260924 07:12:04.607029 13766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:04.607028 13769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:04.607016 13767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:04.608417 13761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:04.608780 13761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:04.609947 13761 hybrid_clock.cc:648] HybridClock initialized: now 1790233924609924 us; error 36 us; skew 500 ppm
Sep 24 07:12:04 dist-test-slave-14ww krb5kdc[13673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:12:04.612391 13761 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 07:12:04.612958 13761 webserver.cc:460] Webserver started at http://127.0.98.61:38083/ using document root <none> and password file <none>
I20260924 07:12:04.613190 13761 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:04.613258 13761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:04.613394 13761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:12:04.614450 13761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-1/data/instance:
uuid: "d3132ae21a25409a8c60eb915cc3f4bf"
format_stamp: "Formatted at 2026-09-24 07:12:04 on dist-test-slave-14ww"
server_key: "d6c2f914b080d85a976c53fdde76ec01"
server_key_iv: "08ac393c870f58217e3281d43597fdc9"
server_key_version: "encryptionkey@0"
I20260924 07:12:04.614830 13761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-1/wal/instance:
uuid: "d3132ae21a25409a8c60eb915cc3f4bf"
format_stamp: "Formatted at 2026-09-24 07:12:04 on dist-test-slave-14ww"
server_key: "d6c2f914b080d85a976c53fdde76ec01"
server_key_iv: "08ac393c870f58217e3281d43597fdc9"
server_key_version: "encryptionkey@0"
I20260924 07:12:04.616137 13761 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 07:12:04.617058 13776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:04.617287 13761 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 07:12:04.617403 13761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-1/wal
uuid: "d3132ae21a25409a8c60eb915cc3f4bf"
format_stamp: "Formatted at 2026-09-24 07:12:04 on dist-test-slave-14ww"
server_key: "d6c2f914b080d85a976c53fdde76ec01"
server_key_iv: "08ac393c870f58217e3281d43597fdc9"
server_key_version: "encryptionkey@0"
I20260924 07:12:04.617492 13761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:04.619436 13754 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32901 } attempt: 2
I20260924 07:12:04.619819 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.619698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32901 (local address 127.0.0.1:46128)
0924 07:12:04.619756 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:04.619793 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 07:12:04.620127 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.620039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32901 (local address 127.0.0.1:46132)
0924 07:12:04.620078 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:04.620104 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 07:12:04.620222 13754 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:32901: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32901: connect: Connection refused (error 111)
I20260924 07:12:04.632308 13761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:04.634382 13761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:04.634547 13761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:04.638515 13761 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:32901
I20260924 07:12:04.638546 13828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:32901 every 8 connection(s)
I20260924 07:12:04.638890 13761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-1/data/info.pb
I20260924 07:12:04.639771 13829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:12:04.641422 13829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45593 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:12:04.644935 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 13761
I20260924 07:12:04.645041 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-1/wal/instance
I20260924 07:12:04.645262 392 external_mini_cluster.cc:1428] Setting key fce8d33e9aaaf270bd4679d7f45cc62b
Sep 24 07:12:04 dist-test-slave-14ww krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:12:04.650678 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.642150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37160 (local address 127.0.98.62:45593)
0924 07:12:04.642237 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:04.642240 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:04.642334 (+ 94us) server_negotiation.cc:407] Connection header received
0924 07:12:04.642772 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:04.642773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:04.642785 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:04.642839 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:04.643415 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.643897 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.644862 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.645107 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.645173 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:04.647848 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:04.647850 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:04.647850 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:04.647859 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:04.649309 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:04.649734 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:04.649736 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:04.649736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:04.649795 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:04.650053 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:04.650054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:04.650054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:04.650163 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:04.650238 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:04.650505 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 07:12:04.650589 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 07:12:04.650840 13831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.642063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45593 (local address 127.0.0.1:37160)
0924 07:12:04.642208 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:04.642252 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:04.642683 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:04.642910 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:04.642913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:04.643157 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:04.643336 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:04.643339 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.643969 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.643970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:04.644778 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:04.644782 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.645198 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.645200 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:04.645347 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:04.645840 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:04.645850 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:04.647722 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:12:04.649408 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:04.649415 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:04.649418 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:04.649642 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:04.649863 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:04.649864 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:04.649864 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:04.649977 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:04.650229 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:12:04.650236 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:12:04.650371 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 07:12:04.650474 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 07:12:04.650618 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":65,"threads_started":1}
I20260924 07:12:04.651371 13829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32901 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 07:12:04 dist-test-slave-14ww krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 07:12:04.658941 13831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.651606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32901 (local address 127.0.0.1:46142)
0924 07:12:04.651653 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:04.651663 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:04.651697 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:04.651953 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:04.651953 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:04.652081 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:04.652120 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:04.652121 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.653090 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.653091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:04.653730 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:04.653732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.654183 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.654183 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:04.654248 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:04.654633 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:04.654637 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:04.655803 (+ 1166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:12:04.657774 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:04.657775 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:04.657775 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:04.657984 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:04.658192 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:04.658193 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:04.658193 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:04.658263 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:04.658514 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:12:04.658514 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:12:04.658561 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 07:12:04.658638 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 07:12:04.658744 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 07:12:04.658957 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.651666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46142 (local address 127.0.98.61:32901)
0924 07:12:04.651808 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:04.651812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:04.651836 (+ 24us) server_negotiation.cc:407] Connection header received
0924 07:12:04.651852 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:04.651853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:04.651863 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:04.651935 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:04.652199 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.653014 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.653811 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.654095 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.654182 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:04.655898 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:04.655905 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:04.655905 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:04.655914 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:04.657671 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:04.658066 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:04.658068 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:04.658069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:04.658121 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:04.658325 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:04.658325 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:04.658325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:04.658448 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:04.658508 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:04.658673 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 07:12:04.658801 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
I20260924 07:12:04.659925 13829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:12:04.660607 13831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.660458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38559 (local address 127.0.0.1:46498)
0924 07:12:04.660516 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:04.660572 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 07:12:04.660815 13778 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38559: connect: Connection refused (error 111)
I20260924 07:12:04.661345 13831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.661232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38559 (local address 127.0.0.1:46500)
0924 07:12:04.661284 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:04.661315 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 07:12:04.661635 13829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38559: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38559: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 07:12:04.672523 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38559
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45593,127.0.98.61:32901,127.0.98.60:38559
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:12:04.710884 13829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } attempt: 1
I20260924 07:12:04.711367 13754 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32901 } attempt: 3
I20260924 07:12:04.711417 13831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.711262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38559 (local address 127.0.0.1:46512)
0924 07:12:04.711327 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:04.711382 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 07:12:04.711967 13831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.711876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38559 (local address 127.0.0.1:46524)
0924 07:12:04.711912 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:04.711934 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 07:12:04.712054 13829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38559: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38559: connect: Connection refused (error 111)
Sep 24 07:12:04 dist-test-slave-14ww krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 07:12:04.718436 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.711699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46146 (local address 127.0.98.61:32901)
0924 07:12:04.711736 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:04.711738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:04.711750 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:12:04.711841 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:04.711842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:04.711853 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:04.711926 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:04.712321 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.712899 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.713620 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.713833 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.713907 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:04.715966 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:04.715967 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:04.715967 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:04.715973 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:04.717393 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:04.717775 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:04.717775 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:04.717776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:04.717827 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:04.718023 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:04.718023 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:04.718023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:04.718119 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:04.718168 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:04.718320 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 07:12:04.718377 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 07:12:04.718499 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.711634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32901 (local address 127.0.0.1:46146)
0924 07:12:04.711694 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:04.711707 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:04.711760 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:04.711931 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:04.711932 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:04.712172 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:04.712233 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:04.712235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.713004 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.713005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:04.713527 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:04.713529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.713916 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.713917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:04.714010 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:04.714465 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:04.714470 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:04.715886 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:12:04.717481 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:04.717482 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:04.717482 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:04.717699 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:04.717895 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:04.717896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:04.717896 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:04.717961 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:04.718177 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:12:04.718178 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:12:04.718229 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 07:12:04.718304 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 07:12:04.718408 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 07:12:04.718992 13754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:12:04.719308 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.719207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38559 (local address 127.0.0.1:46526)
0924 07:12:04.719255 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:04.719286 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 07:12:04.719650 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.719532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38559 (local address 127.0.0.1:46542)
0924 07:12:04.719580 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:04.719620 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 07:12:04.719766 13754 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38559: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38559: connect: Connection refused (error 111)
I20260924 07:12:04.750902 13754 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } attempt: 1
I20260924 07:12:04.751370 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.751240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38559 (local address 127.0.0.1:46558)
0924 07:12:04.751306 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:04.751342 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 07:12:04.751744 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.751659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38559 (local address 127.0.0.1:46572)
0924 07:12:04.751702 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:04.751729 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 07:12:04.751865 13754 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38559: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38559: connect: Connection refused (error 111)
W20260924 07:12:04.755612 13836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:04.755817 13836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:04.755863 13836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:04.757061 13836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:12:04.757120 13836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:04.757146 13836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:04.757193 13836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:12:04.757225 13836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:12:04.758697 13836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45593,127.0.98.61:32901,127.0.98.60:38559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:38559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:12:04.758970 13836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:04.759188 13836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:04.764042 13842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:04.764058 13841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:04.764284 13836 server_base.cc:1034] running on GCE node
W20260924 07:12:04.764117 13844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:04.764592 13836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:04.764891 13836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:04.766078 13836 hybrid_clock.cc:648] HybridClock initialized: now 1790233924766064 us; error 28 us; skew 500 ppm
Sep 24 07:12:04 dist-test-slave-14ww krb5kdc[13673](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:12:04.768433 13836 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 07:12:04.768951 13836 webserver.cc:460] Webserver started at http://127.0.98.60:36865/ using document root <none> and password file <none>
I20260924 07:12:04.769217 13836 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:04.769330 13836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:04.769470 13836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:12:04.770500 13836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-2/data/instance:
uuid: "f61b5b06eb7f4f279b21a6955dc30361"
format_stamp: "Formatted at 2026-09-24 07:12:04 on dist-test-slave-14ww"
server_key: "1b442e6983c044877577dd93765b94a3"
server_key_iv: "004ceb2f354f272543c47802e65861f4"
server_key_version: "encryptionkey@0"
I20260924 07:12:04.770848 13836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-2/wal/instance:
uuid: "f61b5b06eb7f4f279b21a6955dc30361"
format_stamp: "Formatted at 2026-09-24 07:12:04 on dist-test-slave-14ww"
server_key: "1b442e6983c044877577dd93765b94a3"
server_key_iv: "004ceb2f354f272543c47802e65861f4"
server_key_version: "encryptionkey@0"
I20260924 07:12:04.772110 13836 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:12:04.772841 13851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:04.773007 13836 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:12:04.773102 13836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-2/wal
uuid: "f61b5b06eb7f4f279b21a6955dc30361"
format_stamp: "Formatted at 2026-09-24 07:12:04 on dist-test-slave-14ww"
server_key: "1b442e6983c044877577dd93765b94a3"
server_key_iv: "004ceb2f354f272543c47802e65861f4"
server_key_version: "encryptionkey@0"
I20260924 07:12:04.773180 13836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:04.780185 13829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } attempt: 2
I20260924 07:12:04.780547 13831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.780432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38559 (local address 127.0.0.1:46580)
0924 07:12:04.780486 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:04.780517 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 07:12:04.780897 13831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.780802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38559 (local address 127.0.0.1:46590)
0924 07:12:04.780849 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:04.780873 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 07:12:04.780987 13829 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:38559: Network error: Client connection negotiation failed: client connection to 127.0.98.60:38559: connect: Connection refused (error 111)
I20260924 07:12:04.789968 13836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:04.791800 13836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:04.791956 13836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:04.795876 13836 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:38559
I20260924 07:12:04.795908 13903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:38559 every 8 connection(s)
I20260924 07:12:04.796233 13836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-2/data/info.pb
I20260924 07:12:04.796928 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 13836
I20260924 07:12:04.797049 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790233772407728-392-0/minicluster-data/master-2/wal/instance
I20260924 07:12:04.797257 392 external_mini_cluster.cc:1428] Setting key 316e0443a9ea6ead5f5df7b95c71be89
I20260924 07:12:04.797251 13904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:12:04.799217 13904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45593 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:12:04 dist-test-slave-14ww krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 07:12:04 dist-test-slave-14ww krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:12:04.807811 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.799823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37172 (local address 127.0.98.62:45593)
0924 07:12:04.799900 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:04.799902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:04.800044 (+ 142us) server_negotiation.cc:407] Connection header received
0924 07:12:04.800124 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:04.800125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:04.800142 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:04.800205 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:04.800628 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.801140 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.801909 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.802174 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.802243 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:04.804852 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:04.804854 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:04.804854 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:04.804863 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:04.806379 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:04.806804 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:04.806806 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:04.806806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:04.806891 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:04.807197 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:04.807198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:04.807198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:04.807322 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:04.807404 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:04.807630 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 07:12:04.807715 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 07:12:04.807916 13908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.799766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45593 (local address 127.0.0.1:37172)
0924 07:12:04.799940 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:04.799982 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:04.800061 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:04.800225 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:04.800229 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:04.800472 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:04.800535 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:04.800537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.801222 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.801223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:04.801808 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:04.801810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.802262 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.802263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:04.802349 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:04.802728 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:04.802739 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:04.804762 (+ 2023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:12:04.806454 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:04.806460 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:04.806462 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:04.806708 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:04.806983 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:04.806984 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:04.806984 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:04.807125 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:04.807399 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:12:04.807404 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:12:04.807459 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 07:12:04.807528 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 07:12:04.807684 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":87,"threads_started":1}
I20260924 07:12:04.808058 13907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.797755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46604 (local address 127.0.98.60:38559)
0924 07:12:04.797955 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:04.797964 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:04.798041 (+ 77us) server_negotiation.cc:407] Connection header received
0924 07:12:04.798574 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:04.798581 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:04.798631 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:04.798772 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:04.799442 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.800521 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.801312 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.801715 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.801846 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:04.804316 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:04.804322 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:04.804322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:04.804329 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:04.806312 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:04.806811 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:04.806814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:04.806815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:04.806930 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:04.807123 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:04.807123 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:04.807124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:04.807476 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:04.807715 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:04.807779 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 07:12:04.807855 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":84,"threads_started":1}
I20260924 07:12:04.808609 13904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32901 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:12:04 dist-test-slave-14ww krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 24 07:12:04 dist-test-slave-14ww krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:12:04.815174 13908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.808932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32901 (local address 127.0.0.1:46148)
0924 07:12:04.808963 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:04.808973 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:04.809010 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:04.809169 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:04.809170 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:04.809314 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:04.809363 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:04.809365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.810080 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.810081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:04.810673 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:04.810675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.811007 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.811008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:04.811063 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:04.811483 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:04.811487 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:04.812662 (+ 1175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:12:04.814183 (+ 1521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:04.814184 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:04.814184 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:04.814343 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:04.814556 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:04.814557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:04.814557 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:04.814623 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:04.814866 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:12:04.814867 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:12:04.814906 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 07:12:04.815000 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 07:12:04.815087 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
I20260924 07:12:04.815210 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.809008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46148 (local address 127.0.98.61:32901)
0924 07:12:04.809073 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:04.809075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:04.809084 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:12:04.809109 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:04.809110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:04.809119 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:04.809168 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:04.809447 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.810000 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.810752 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.810941 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.810990 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:04.812765 (+ 1775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:04.812766 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:04.812767 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:04.812773 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:04.814102 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:04.814426 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:04.814427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:04.814427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:04.814492 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:04.814686 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:04.814687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:04.814687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:04.814801 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:04.814873 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:04.815026 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 07:12:04.815094 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 07:12:04.815750 13904 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:12:04.817026 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.809232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37182 (local address 127.0.98.62:45593)
0924 07:12:04.809266 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:04.809267 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:04.809272 (+ 5us) server_negotiation.cc:407] Connection header received
0924 07:12:04.809287 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:04.809288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:04.809294 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:04.809331 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:04.809796 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.810324 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.811134 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.811297 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.811371 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:04.813836 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:04.813837 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:04.813837 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:04.813843 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:04.815246 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:04.815564 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:04.815565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:04.815566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:04.815625 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:04.816072 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:04.816073 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:04.816074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:04.816208 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:04.816830 (+ 622us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:04.816871 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 07:12:04.816942 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Sep 24 07:12:04 dist-test-slave-14ww krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 07:12:04 dist-test-slave-14ww krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 07:12:04.822316 13908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.815951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38559 (local address 127.0.0.1:46612)
0924 07:12:04.815996 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:04.816006 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:04.816043 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:04.816180 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:04.816181 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:04.816319 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:04.816347 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:04.816349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.817273 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.817274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:04.817968 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:04.817971 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.818343 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.818344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:04.818389 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:04.818767 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:04.818771 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:04.819842 (+ 1071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:12:04.821374 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:04.821374 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:04.821375 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:04.821566 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:04.821761 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:04.821762 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:04.821762 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:04.821826 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:04.822074 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:12:04.822075 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:12:04.822118 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 07:12:04.822186 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 07:12:04.822262 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 07:12:04.822423 13907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.815996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46612 (local address 127.0.98.60:38559)
0924 07:12:04.816024 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:04.816025 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:04.816030 (+ 5us) server_negotiation.cc:407] Connection header received
0924 07:12:04.816107 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:04.816108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:04.816116 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:04.816179 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:04.816426 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.817192 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.818061 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.818274 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.818346 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:04.819915 (+ 1569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:04.819916 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:04.819917 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:04.819923 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:04.821282 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:04.821644 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:04.821644 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:04.821645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:04.821693 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:04.821905 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:04.821906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:04.821906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:04.822007 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:04.822057 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:04.822211 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 07:12:04.822331 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":14}
I20260924 07:12:04.823587 13904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f61b5b06eb7f4f279b21a6955dc30361: Bootstrap starting.
I20260924 07:12:04.824384 13904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f61b5b06eb7f4f279b21a6955dc30361: Neither blocks nor log segments found. Creating new log.
I20260924 07:12:04.824677 13904 log.cc:824] T 00000000000000000000000000000000 P f61b5b06eb7f4f279b21a6955dc30361: Log is configured to *not* fsync() on all Append() calls
I20260924 07:12:04.825075 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.817725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46164 (local address 127.0.98.61:32901)
0924 07:12:04.817767 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:04.817768 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:04.817915 (+ 147us) server_negotiation.cc:407] Connection header received
0924 07:12:04.817985 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:04.817986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:04.817995 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:04.818050 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:04.818497 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.819037 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.819708 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.819926 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.819997 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:04.822540 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:04.822541 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:04.822542 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:04.822550 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:04.823982 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:04.824321 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:04.824322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:04.824322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:04.824375 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:04.824562 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:04.824563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:04.824563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:04.824664 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:04.824862 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:04.824901 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 07:12:04.824971 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 07:12:04.825448 13904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f61b5b06eb7f4f279b21a6955dc30361: No bootstrap required, opened a new log
I20260924 07:12:04.826735 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:12:04.827003 13754 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } attempt: 2
I20260924 07:12:04.827306 13904 raft_consensus.cc:348] T 00000000000000000000000000000000 P f61b5b06eb7f4f279b21a6955dc30361 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68aa88f30b5947379e2c90835a39ba78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45593 } } peers { permanent_uuid: "d3132ae21a25409a8c60eb915cc3f4bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32901 } } peers { permanent_uuid: "f61b5b06eb7f4f279b21a6955dc30361" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } }
I20260924 07:12:04.827503 13904 raft_consensus.cc:374] T 00000000000000000000000000000000 P f61b5b06eb7f4f279b21a6955dc30361 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:12:04.827592 13904 raft_consensus.cc:729] T 00000000000000000000000000000000 P f61b5b06eb7f4f279b21a6955dc30361 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f61b5b06eb7f4f279b21a6955dc30361, State: Initialized, Role: FOLLOWER
I20260924 07:12:04.827749 13904 consensus_queue.cc:260] T 00000000000000000000000000000000 P f61b5b06eb7f4f279b21a6955dc30361 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68aa88f30b5947379e2c90835a39ba78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45593 } } peers { permanent_uuid: "d3132ae21a25409a8c60eb915cc3f4bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32901 } } peers { permanent_uuid: "f61b5b06eb7f4f279b21a6955dc30361" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } }
I20260924 07:12:04.828091 13904 sys_catalog.cc:564] T 00000000000000000000000000000000 P f61b5b06eb7f4f279b21a6955dc30361 [sys.catalog]: configured and running, proceeding with master startup.
Sep 24 07:12:04 dist-test-slave-14ww krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 07:12:04.830123 13836 master_runner.cc:186] Error getting master registration for 127.0.98.62:45593: OK, instance_id { permanent_uuid: "68aa88f30b5947379e2c90835a39ba78" instance_seqno: 1790233924486262 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 07:12:04.833247 13836 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 07:12:04.834218 13929 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f61b5b06eb7f4f279b21a6955dc30361: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:12:04.834329 13929 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:12:04.834641 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.827223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38559 (local address 127.0.0.1:46626)
0924 07:12:04.827266 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:04.827276 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:04.827309 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:04.827448 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:04.827449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:04.827610 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:04.827652 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:04.827654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.828497 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.828498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:04.829354 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:04.829357 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.829843 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.829844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:04.829893 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:04.830301 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:04.830307 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:04.831702 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:12:04.833506 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:04.833506 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:04.833507 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:04.833730 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:04.833949 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:04.833950 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:04.833951 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:04.834024 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:04.834334 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:12:04.834335 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:12:04.834381 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 07:12:04.834447 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 07:12:04.834549 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 07:12:04.834908 13907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.827283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46626 (local address 127.0.98.60:38559)
0924 07:12:04.827317 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:04.827319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:04.827329 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:12:04.827367 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:04.827368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:04.827375 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:04.827442 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:04.827736 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.828384 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.829471 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.829761 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.830036 (+ 275us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:04.831807 (+ 1771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:04.831809 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:04.831809 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:04.831816 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:04.833401 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:04.833808 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:04.833809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:04.833809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:04.833869 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:04.834104 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:04.834105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:04.834105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:04.834244 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:04.834306 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:04.834759 (+ 453us) server_negotiation.cc:299] Negotiation successful
0924 07:12:04.834831 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 07:12:04.835772 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.828154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46178 (local address 127.0.98.61:32901)
0924 07:12:04.828190 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:04.828192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:04.828295 (+ 103us) server_negotiation.cc:407] Connection header received
0924 07:12:04.828406 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:04.828407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:04.828418 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:04.828447 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:04.829345 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.830058 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.831058 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.831345 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.831383 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:04.832923 (+ 1540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:04.832924 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:04.832925 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:04.832932 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:04.834566 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:04.834945 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:04.834946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:04.834946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:04.835003 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:04.835242 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:04.835243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:04.835244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:04.835364 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:04.835426 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:04.835614 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 07:12:04.835684 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 07:12:04.836393 13754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68aa88f30b5947379e2c90835a39ba78: Bootstrap starting.
I20260924 07:12:04.836969 13914 sys_catalog.cc:455] T 00000000000000000000000000000000 P f61b5b06eb7f4f279b21a6955dc30361 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68aa88f30b5947379e2c90835a39ba78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45593 } } peers { permanent_uuid: "d3132ae21a25409a8c60eb915cc3f4bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32901 } } peers { permanent_uuid: "f61b5b06eb7f4f279b21a6955dc30361" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } } }
I20260924 07:12:04.837155 13914 sys_catalog.cc:458] T 00000000000000000000000000000000 P f61b5b06eb7f4f279b21a6955dc30361 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:12:04.837347 13754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68aa88f30b5947379e2c90835a39ba78: Neither blocks nor log segments found. Creating new log.
I20260924 07:12:04.837704 13754 log.cc:824] T 00000000000000000000000000000000 P 68aa88f30b5947379e2c90835a39ba78: Log is configured to *not* fsync() on all Append() calls
I20260924 07:12:04.839102 13754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68aa88f30b5947379e2c90835a39ba78: No bootstrap required, opened a new log
I20260924 07:12:04.840420 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.828453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46628 (local address 127.0.98.60:38559)
0924 07:12:04.829087 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:04.829089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:04.829096 (+ 7us) server_negotiation.cc:407] Connection header received
0924 07:12:04.829115 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:04.829116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:04.829125 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:04.829182 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:04.829860 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.830908 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.835295 (+ 4387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.835579 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.835645 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:04.837280 (+ 1635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:04.837281 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:04.837281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:04.837288 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:04.839056 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:04.839446 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:04.839447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:04.839448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:04.839514 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:04.839704 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:04.839704 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:04.839705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:04.839816 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:04.839934 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:04.840238 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 07:12:04.840320 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":78,"threads_started":1}
I20260924 07:12:04.841581 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.828699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37190 (local address 127.0.98.62:45593)
0924 07:12:04.828755 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:04.828757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:04.832000 (+ 3243us) server_negotiation.cc:407] Connection header received
0924 07:12:04.832357 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:04.832358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:04.832369 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:04.832402 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:04.833955 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.834665 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.836294 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.836555 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.836607 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:04.838424 (+ 1817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:04.838425 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:04.838425 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:04.838432 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:04.840089 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:04.840502 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:04.840503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:04.840503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:04.840557 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:04.840944 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:04.840944 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:04.840945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:04.841063 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:04.841139 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:04.841417 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 07:12:04.841496 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 07:12:04.841897 13754 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68aa88f30b5947379e2c90835a39ba78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68aa88f30b5947379e2c90835a39ba78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45593 } } peers { permanent_uuid: "d3132ae21a25409a8c60eb915cc3f4bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32901 } } peers { permanent_uuid: "f61b5b06eb7f4f279b21a6955dc30361" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } }
I20260924 07:12:04.842051 13754 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68aa88f30b5947379e2c90835a39ba78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:12:04.842097 13754 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68aa88f30b5947379e2c90835a39ba78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68aa88f30b5947379e2c90835a39ba78, State: Initialized, Role: FOLLOWER
I20260924 07:12:04.842216 13754 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68aa88f30b5947379e2c90835a39ba78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68aa88f30b5947379e2c90835a39ba78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45593 } } peers { permanent_uuid: "d3132ae21a25409a8c60eb915cc3f4bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32901 } } peers { permanent_uuid: "f61b5b06eb7f4f279b21a6955dc30361" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } }
I20260924 07:12:04.842490 13932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68aa88f30b5947379e2c90835a39ba78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68aa88f30b5947379e2c90835a39ba78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45593 } } peers { permanent_uuid: "d3132ae21a25409a8c60eb915cc3f4bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32901 } } peers { permanent_uuid: "f61b5b06eb7f4f279b21a6955dc30361" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } } }
I20260924 07:12:04.842545 13754 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68aa88f30b5947379e2c90835a39ba78 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:12:04.842571 13932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68aa88f30b5947379e2c90835a39ba78 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 07:12:04.844167 13943 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 68aa88f30b5947379e2c90835a39ba78: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:12:04.844267 13943 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:12:04.845026 13686 master_runner.cc:186] Error getting master registration for 127.0.98.61:32901: OK, instance_id { permanent_uuid: "d3132ae21a25409a8c60eb915cc3f4bf" instance_seqno: 1790233924637383 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 07:12:04.847302 13686 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 07:12:04.872130 13829 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } attempt: 3
Sep 24 07:12:04 dist-test-slave-14ww krb5kdc[13673](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 07:12:04.880625 13831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.872408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:38559 (local address 127.0.0.1:46632)
0924 07:12:04.872459 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:04.872470 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:04.872506 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:04.872689 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:04.872690 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:04.872915 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:04.872967 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:04.872968 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.873906 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.873907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:04.874725 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:04.874727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.875278 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.875279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:04.875346 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:04.875864 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:04.875869 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:04.877609 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:12:04.879507 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:04.879507 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:04.879508 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:04.879719 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:04.879950 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:04.879951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:04.879951 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:04.880016 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:04.880294 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:12:04.880295 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:12:04.880351 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 07:12:04.880424 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 07:12:04.880520 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 07:12:04.880637 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:04.872527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46632 (local address 127.0.98.60:38559)
0924 07:12:04.872567 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:04.872569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:04.872577 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:12:04.872598 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:04.872598 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:04.872608 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:04.872671 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:04.873045 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.873811 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.874814 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:04.875182 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:04.875524 (+ 342us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:04.877695 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:04.877697 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:04.877697 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:04.877705 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:04.879416 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:04.879798 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:04.879799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:04.879800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:04.879873 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:04.880085 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:04.880086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:04.880086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:04.880217 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:04.880285 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:04.880462 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 07:12:04.880543 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 07:12:04.882644 13829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3132ae21a25409a8c60eb915cc3f4bf: Bootstrap starting.
I20260924 07:12:04.883412 13829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3132ae21a25409a8c60eb915cc3f4bf: Neither blocks nor log segments found. Creating new log.
I20260924 07:12:04.883741 13829 log.cc:824] T 00000000000000000000000000000000 P d3132ae21a25409a8c60eb915cc3f4bf: Log is configured to *not* fsync() on all Append() calls
I20260924 07:12:04.884631 13829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3132ae21a25409a8c60eb915cc3f4bf: No bootstrap required, opened a new log
I20260924 07:12:04.886883 13829 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3132ae21a25409a8c60eb915cc3f4bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68aa88f30b5947379e2c90835a39ba78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45593 } } peers { permanent_uuid: "d3132ae21a25409a8c60eb915cc3f4bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32901 } } peers { permanent_uuid: "f61b5b06eb7f4f279b21a6955dc30361" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } }
I20260924 07:12:04.887034 13829 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3132ae21a25409a8c60eb915cc3f4bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:12:04.887068 13829 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3132ae21a25409a8c60eb915cc3f4bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3132ae21a25409a8c60eb915cc3f4bf, State: Initialized, Role: FOLLOWER
I20260924 07:12:04.887219 13829 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3132ae21a25409a8c60eb915cc3f4bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68aa88f30b5947379e2c90835a39ba78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45593 } } peers { permanent_uuid: "d3132ae21a25409a8c60eb915cc3f4bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32901 } } peers { permanent_uuid: "f61b5b06eb7f4f279b21a6955dc30361" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } }
I20260924 07:12:04.887490 13946 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3132ae21a25409a8c60eb915cc3f4bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68aa88f30b5947379e2c90835a39ba78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45593 } } peers { permanent_uuid: "d3132ae21a25409a8c60eb915cc3f4bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32901 } } peers { permanent_uuid: "f61b5b06eb7f4f279b21a6955dc30361" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } } }
I20260924 07:12:04.887550 13829 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3132ae21a25409a8c60eb915cc3f4bf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:12:04.887641 13946 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3132ae21a25409a8c60eb915cc3f4bf [sys.catalog]: This master's current role is: FOLLOWER
W20260924 07:12:04.889487 13957 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d3132ae21a25409a8c60eb915cc3f4bf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:12:04.889588 13957 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:12:04.891480 13761 master_runner.cc:263] No leader master found from master d3132ae21a25409a8c60eb915cc3f4bf
I20260924 07:12:04.891526 13761 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 07:12:04.984329 13707 tablet.cc:2369] T 00000000000000000000000000000000 P 68aa88f30b5947379e2c90835a39ba78: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:12:04.997905 13932 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68aa88f30b5947379e2c90835a39ba78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:12:04.998027 13932 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68aa88f30b5947379e2c90835a39ba78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68aa88f30b5947379e2c90835a39ba78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45593 } } peers { permanent_uuid: "d3132ae21a25409a8c60eb915cc3f4bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32901 } } peers { permanent_uuid: "f61b5b06eb7f4f279b21a6955dc30361" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } }
I20260924 07:12:04.998291 13932 leader_election.cc:288] T 00000000000000000000000000000000 P 68aa88f30b5947379e2c90835a39ba78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3132ae21a25409a8c60eb915cc3f4bf (127.0.98.61:32901), f61b5b06eb7f4f279b21a6955dc30361 (127.0.98.60:38559)
I20260924 07:12:04.998528 13878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "68aa88f30b5947379e2c90835a39ba78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f61b5b06eb7f4f279b21a6955dc30361" is_pre_election: true
I20260924 07:12:04.998591 13803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "68aa88f30b5947379e2c90835a39ba78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3132ae21a25409a8c60eb915cc3f4bf" is_pre_election: true
I20260924 07:12:04.998696 13878 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f61b5b06eb7f4f279b21a6955dc30361 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68aa88f30b5947379e2c90835a39ba78 in term 0.
I20260924 07:12:04.998719 13803 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d3132ae21a25409a8c60eb915cc3f4bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68aa88f30b5947379e2c90835a39ba78 in term 0.
I20260924 07:12:04.998914 13703 leader_election.cc:302] T 00000000000000000000000000000000 P 68aa88f30b5947379e2c90835a39ba78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68aa88f30b5947379e2c90835a39ba78, d3132ae21a25409a8c60eb915cc3f4bf; no voters:
I20260924 07:12:04.999028 13932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68aa88f30b5947379e2c90835a39ba78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:12:04.999081 13932 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68aa88f30b5947379e2c90835a39ba78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:12:04.999114 13932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68aa88f30b5947379e2c90835a39ba78 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:12:04.999800 13932 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68aa88f30b5947379e2c90835a39ba78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68aa88f30b5947379e2c90835a39ba78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45593 } } peers { permanent_uuid: "d3132ae21a25409a8c60eb915cc3f4bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32901 } } peers { permanent_uuid: "f61b5b06eb7f4f279b21a6955dc30361" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } }
I20260924 07:12:04.999943 13932 leader_election.cc:288] T 00000000000000000000000000000000 P 68aa88f30b5947379e2c90835a39ba78 [CANDIDATE]: Term 1 election: Requested vote from peers d3132ae21a25409a8c60eb915cc3f4bf (127.0.98.61:32901), f61b5b06eb7f4f279b21a6955dc30361 (127.0.98.60:38559)
I20260924 07:12:05.000124 13878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "68aa88f30b5947379e2c90835a39ba78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f61b5b06eb7f4f279b21a6955dc30361"
I20260924 07:12:05.000149 13803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "68aa88f30b5947379e2c90835a39ba78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3132ae21a25409a8c60eb915cc3f4bf"
I20260924 07:12:05.000197 13878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f61b5b06eb7f4f279b21a6955dc30361 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:12:05.000208 13803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3132ae21a25409a8c60eb915cc3f4bf [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:12:05.001056 13878 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f61b5b06eb7f4f279b21a6955dc30361 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68aa88f30b5947379e2c90835a39ba78 in term 1.
I20260924 07:12:05.001056 13803 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d3132ae21a25409a8c60eb915cc3f4bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68aa88f30b5947379e2c90835a39ba78 in term 1.
I20260924 07:12:05.001219 13703 leader_election.cc:302] T 00000000000000000000000000000000 P 68aa88f30b5947379e2c90835a39ba78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68aa88f30b5947379e2c90835a39ba78, d3132ae21a25409a8c60eb915cc3f4bf; no voters:
I20260924 07:12:05.001317 13932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68aa88f30b5947379e2c90835a39ba78 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:12:05.001466 13932 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68aa88f30b5947379e2c90835a39ba78 [term 1 LEADER]: Becoming Leader. State: Replica: 68aa88f30b5947379e2c90835a39ba78, State: Running, Role: LEADER
I20260924 07:12:05.001549 13932 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68aa88f30b5947379e2c90835a39ba78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68aa88f30b5947379e2c90835a39ba78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45593 } } peers { permanent_uuid: "d3132ae21a25409a8c60eb915cc3f4bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32901 } } peers { permanent_uuid: "f61b5b06eb7f4f279b21a6955dc30361" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } }
I20260924 07:12:05.001928 13959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68aa88f30b5947379e2c90835a39ba78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68aa88f30b5947379e2c90835a39ba78. Latest consensus state: current_term: 1 leader_uuid: "68aa88f30b5947379e2c90835a39ba78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68aa88f30b5947379e2c90835a39ba78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45593 } } peers { permanent_uuid: "d3132ae21a25409a8c60eb915cc3f4bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32901 } } peers { permanent_uuid: "f61b5b06eb7f4f279b21a6955dc30361" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } } }
I20260924 07:12:05.002054 13959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68aa88f30b5947379e2c90835a39ba78 [sys.catalog]: This master's current role is: LEADER
I20260924 07:12:05.002271 13961 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:12:05.002481 13961 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:12:05.003878 13878 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f61b5b06eb7f4f279b21a6955dc30361 [term 1 FOLLOWER]: Refusing update from remote peer 68aa88f30b5947379e2c90835a39ba78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:12:05.003876 13803 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d3132ae21a25409a8c60eb915cc3f4bf [term 1 FOLLOWER]: Refusing update from remote peer 68aa88f30b5947379e2c90835a39ba78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:12:05.004161 13959 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 68aa88f30b5947379e2c90835a39ba78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3132ae21a25409a8c60eb915cc3f4bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:12:05.004335 13959 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 68aa88f30b5947379e2c90835a39ba78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f61b5b06eb7f4f279b21a6955dc30361" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:12:05.005406 13946 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3132ae21a25409a8c60eb915cc3f4bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68aa88f30b5947379e2c90835a39ba78. Latest consensus state: current_term: 1 leader_uuid: "68aa88f30b5947379e2c90835a39ba78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68aa88f30b5947379e2c90835a39ba78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45593 } } peers { permanent_uuid: "d3132ae21a25409a8c60eb915cc3f4bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32901 } } peers { permanent_uuid: "f61b5b06eb7f4f279b21a6955dc30361" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } } }
I20260924 07:12:05.005499 13946 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3132ae21a25409a8c60eb915cc3f4bf [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:12:05.005538 13914 sys_catalog.cc:455] T 00000000000000000000000000000000 P f61b5b06eb7f4f279b21a6955dc30361 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68aa88f30b5947379e2c90835a39ba78. Latest consensus state: current_term: 1 leader_uuid: "68aa88f30b5947379e2c90835a39ba78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68aa88f30b5947379e2c90835a39ba78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45593 } } peers { permanent_uuid: "d3132ae21a25409a8c60eb915cc3f4bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32901 } } peers { permanent_uuid: "f61b5b06eb7f4f279b21a6955dc30361" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } } }
I20260924 07:12:05.005627 13914 sys_catalog.cc:458] T 00000000000000000000000000000000 P f61b5b06eb7f4f279b21a6955dc30361 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:12:05.006421 13962 mvcc.cc:205] Tried to move back new op lower bound from 7332798156813541376 to 7332798156806918144. Current Snapshot: MvccSnapshot[applied={T|T < 7332798156813541376}]
I20260924 07:12:05.006488 13959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68aa88f30b5947379e2c90835a39ba78 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "68aa88f30b5947379e2c90835a39ba78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68aa88f30b5947379e2c90835a39ba78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45593 } } peers { permanent_uuid: "d3132ae21a25409a8c60eb915cc3f4bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32901 } } peers { permanent_uuid: "f61b5b06eb7f4f279b21a6955dc30361" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } } }
I20260924 07:12:05.006597 13959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68aa88f30b5947379e2c90835a39ba78 [sys.catalog]: This master's current role is: LEADER
I20260924 07:12:05.007040 13946 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3132ae21a25409a8c60eb915cc3f4bf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "68aa88f30b5947379e2c90835a39ba78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68aa88f30b5947379e2c90835a39ba78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45593 } } peers { permanent_uuid: "d3132ae21a25409a8c60eb915cc3f4bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32901 } } peers { permanent_uuid: "f61b5b06eb7f4f279b21a6955dc30361" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } } }
I20260924 07:12:05.007115 13946 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3132ae21a25409a8c60eb915cc3f4bf [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:12:05.007349 13963 mvcc.cc:205] Tried to move back new op lower bound from 7332798156813541376 to 7332798156806918144. Current Snapshot: MvccSnapshot[applied={T|T < 7332798156813541376}]
I20260924 07:12:05.007536 13961 catalog_manager.cc:1347] Generated new cluster ID: e2e2db8b90834eeda2a85c17de7938f7
I20260924 07:12:05.007604 13961 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:12:05.007628 13959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68aa88f30b5947379e2c90835a39ba78 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "68aa88f30b5947379e2c90835a39ba78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68aa88f30b5947379e2c90835a39ba78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45593 } } peers { permanent_uuid: "d3132ae21a25409a8c60eb915cc3f4bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32901 } } peers { permanent_uuid: "f61b5b06eb7f4f279b21a6955dc30361" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } } }
I20260924 07:12:05.007689 13959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68aa88f30b5947379e2c90835a39ba78 [sys.catalog]: This master's current role is: LEADER
I20260924 07:12:05.008103 13964 mvcc.cc:205] Tried to move back new op lower bound from 7332798156813541376 to 7332798156806918144. Current Snapshot: MvccSnapshot[applied={T|T < 7332798156813541376}]
I20260924 07:12:05.008208 13914 sys_catalog.cc:455] T 00000000000000000000000000000000 P f61b5b06eb7f4f279b21a6955dc30361 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "68aa88f30b5947379e2c90835a39ba78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68aa88f30b5947379e2c90835a39ba78" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45593 } } peers { permanent_uuid: "d3132ae21a25409a8c60eb915cc3f4bf" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32901 } } peers { permanent_uuid: "f61b5b06eb7f4f279b21a6955dc30361" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 38559 } } }
I20260924 07:12:05.008267 13914 sys_catalog.cc:458] T 00000000000000000000000000000000 P f61b5b06eb7f4f279b21a6955dc30361 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:12:05.014626 13961 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:12:05.015318 13961 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:12:05.025456 13961 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68aa88f30b5947379e2c90835a39ba78: Generated new TSK 0
I20260924 07:12:05.025643 13961 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:12:05.834838 13929 catalog_manager.cc:1259] Loaded cluster ID: e2e2db8b90834eeda2a85c17de7938f7
I20260924 07:12:05.834918 13929 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f61b5b06eb7f4f279b21a6955dc30361: loading cluster ID for follower catalog manager: success
I20260924 07:12:05.835700 13929 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f61b5b06eb7f4f279b21a6955dc30361: acquiring CA information for follower catalog manager: success
I20260924 07:12:05.835948 13929 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f61b5b06eb7f4f279b21a6955dc30361: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 07:12:05.890084 13957 catalog_manager.cc:1259] Loaded cluster ID: e2e2db8b90834eeda2a85c17de7938f7
I20260924 07:12:05.890166 13957 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d3132ae21a25409a8c60eb915cc3f4bf: loading cluster ID for follower catalog manager: success
I20260924 07:12:05.890929 13957 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d3132ae21a25409a8c60eb915cc3f4bf: acquiring CA information for follower catalog manager: success
I20260924 07:12:05.891184 13957 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d3132ae21a25409a8c60eb915cc3f4bf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 07:12:06.077342 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 13686
I20260924 07:12:06.082690 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 13761
I20260924 07:12:06.087474 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 13836
2026-09-24T07:12:06Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2891 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:12:06 dist-test-slave-14ww krb5kdc[13977](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:12:06 dist-test-slave-14ww krb5kdc[13977](info): set up 2 sockets
Sep 24 07:12:06 dist-test-slave-14ww krb5kdc[13977](info): commencing operation
krb5kdc: starting...
W20260924 07:12:07.128943 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:12:07 dist-test-slave-14ww krb5kdc[13977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:12:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:12:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:12:07.245502 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46527
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46527
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:12:07.323916 13990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:07.324149 13990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:07.324187 13990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:07.325392 13990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:12:07.325465 13990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:07.325489 13990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:07.325517 13990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:12:07.325546 13990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:12:07.326973 13990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:12:07.327248 13990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:07.327474 13990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:07.332114 13998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:07.332134 13996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:07.332156 13995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:07.332453 13990 server_base.cc:1034] running on GCE node
I20260924 07:12:07.332778 13990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:07.333068 13990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:07.334235 13990 hybrid_clock.cc:648] HybridClock initialized: now 1790233927334233 us; error 41 us; skew 500 ppm
Sep 24 07:12:07 dist-test-slave-14ww krb5kdc[13977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:12:07.336530 13990 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:12:07.336984 13990 webserver.cc:460] Webserver started at http://127.0.98.62:45871/ using document root <none> and password file <none>
I20260924 07:12:07.337224 13990 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:07.337291 13990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:07.337438 13990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:12:07.338469 13990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "e85c9a5745e34cb7b7d89d666a9087d6"
format_stamp: "Formatted at 2026-09-24 07:12:07 on dist-test-slave-14ww"
server_key: "4eb8300d132efc88c18cbc0f50136a0e"
server_key_iv: "ed31771cd1fd3eeadc249f7d2536f2b9"
server_key_version: "encryptionkey@0"
I20260924 07:12:07.338822 13990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "e85c9a5745e34cb7b7d89d666a9087d6"
format_stamp: "Formatted at 2026-09-24 07:12:07 on dist-test-slave-14ww"
server_key: "4eb8300d132efc88c18cbc0f50136a0e"
server_key_iv: "ed31771cd1fd3eeadc249f7d2536f2b9"
server_key_version: "encryptionkey@0"
I20260924 07:12:07.340143 13990 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:12:07.340884 14005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:07.341061 13990 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:07.341156 13990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "e85c9a5745e34cb7b7d89d666a9087d6"
format_stamp: "Formatted at 2026-09-24 07:12:07 on dist-test-slave-14ww"
server_key: "4eb8300d132efc88c18cbc0f50136a0e"
server_key_iv: "ed31771cd1fd3eeadc249f7d2536f2b9"
server_key_version: "encryptionkey@0"
I20260924 07:12:07.341240 13990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:07.349997 13990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:07.351740 13990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:07.351900 13990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:07.355840 13990 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46527
I20260924 07:12:07.355870 14057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46527 every 8 connection(s)
I20260924 07:12:07.356209 13990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:12:07.357132 14058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:12:07.359313 14058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e85c9a5745e34cb7b7d89d666a9087d6: Bootstrap starting.
I20260924 07:12:07.359578 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 13990
I20260924 07:12:07.359685 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:12:07.359884 392 external_mini_cluster.cc:1428] Setting key 64921a273904d6a2eba696257a394024
I20260924 07:12:07.359957 14058 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e85c9a5745e34cb7b7d89d666a9087d6: Neither blocks nor log segments found. Creating new log.
I20260924 07:12:07.360293 14058 log.cc:824] T 00000000000000000000000000000000 P e85c9a5745e34cb7b7d89d666a9087d6: Log is configured to *not* fsync() on all Append() calls
I20260924 07:12:07.361016 14058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e85c9a5745e34cb7b7d89d666a9087d6: No bootstrap required, opened a new log
I20260924 07:12:07.363240 14058 raft_consensus.cc:348] T 00000000000000000000000000000000 P e85c9a5745e34cb7b7d89d666a9087d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e85c9a5745e34cb7b7d89d666a9087d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46527 } }
I20260924 07:12:07.363406 14058 raft_consensus.cc:374] T 00000000000000000000000000000000 P e85c9a5745e34cb7b7d89d666a9087d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:12:07.363440 14058 raft_consensus.cc:729] T 00000000000000000000000000000000 P e85c9a5745e34cb7b7d89d666a9087d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e85c9a5745e34cb7b7d89d666a9087d6, State: Initialized, Role: FOLLOWER
I20260924 07:12:07.363548 14058 consensus_queue.cc:260] T 00000000000000000000000000000000 P e85c9a5745e34cb7b7d89d666a9087d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e85c9a5745e34cb7b7d89d666a9087d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46527 } }
I20260924 07:12:07.363613 14058 raft_consensus.cc:388] T 00000000000000000000000000000000 P e85c9a5745e34cb7b7d89d666a9087d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:12:07.363641 14058 raft_consensus.cc:482] T 00000000000000000000000000000000 P e85c9a5745e34cb7b7d89d666a9087d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:12:07.363683 14058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e85c9a5745e34cb7b7d89d666a9087d6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:12:07.364269 14058 raft_consensus.cc:504] T 00000000000000000000000000000000 P e85c9a5745e34cb7b7d89d666a9087d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e85c9a5745e34cb7b7d89d666a9087d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46527 } }
I20260924 07:12:07.364391 14058 leader_election.cc:302] T 00000000000000000000000000000000 P e85c9a5745e34cb7b7d89d666a9087d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e85c9a5745e34cb7b7d89d666a9087d6; no voters:
I20260924 07:12:07.364596 14058 leader_election.cc:288] T 00000000000000000000000000000000 P e85c9a5745e34cb7b7d89d666a9087d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:12:07.364626 14063 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e85c9a5745e34cb7b7d89d666a9087d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:12:07.364794 14063 raft_consensus.cc:686] T 00000000000000000000000000000000 P e85c9a5745e34cb7b7d89d666a9087d6 [term 1 LEADER]: Becoming Leader. State: Replica: e85c9a5745e34cb7b7d89d666a9087d6, State: Running, Role: LEADER
I20260924 07:12:07.364886 14063 consensus_queue.cc:237] T 00000000000000000000000000000000 P e85c9a5745e34cb7b7d89d666a9087d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e85c9a5745e34cb7b7d89d666a9087d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46527 } }
I20260924 07:12:07.365227 14058 sys_catalog.cc:564] T 00000000000000000000000000000000 P e85c9a5745e34cb7b7d89d666a9087d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:12:07.365197 14064 sys_catalog.cc:455] T 00000000000000000000000000000000 P e85c9a5745e34cb7b7d89d666a9087d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e85c9a5745e34cb7b7d89d666a9087d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e85c9a5745e34cb7b7d89d666a9087d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46527 } } }
I20260924 07:12:07.365330 14064 sys_catalog.cc:458] T 00000000000000000000000000000000 P e85c9a5745e34cb7b7d89d666a9087d6 [sys.catalog]: This master's current role is: LEADER
I20260924 07:12:07.365185 14063 sys_catalog.cc:455] T 00000000000000000000000000000000 P e85c9a5745e34cb7b7d89d666a9087d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e85c9a5745e34cb7b7d89d666a9087d6. Latest consensus state: current_term: 1 leader_uuid: "e85c9a5745e34cb7b7d89d666a9087d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e85c9a5745e34cb7b7d89d666a9087d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46527 } } }
I20260924 07:12:07.365538 14067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:12:07.365561 14063 sys_catalog.cc:458] T 00000000000000000000000000000000 P e85c9a5745e34cb7b7d89d666a9087d6 [sys.catalog]: This master's current role is: LEADER
I20260924 07:12:07.365993 14067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 07:12:07 dist-test-slave-14ww krb5kdc[13977](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:12:07.367677 14067 catalog_manager.cc:1347] Generated new cluster ID: 92430fa2d63c41ed868381b1bfe982b9
I20260924 07:12:07.367729 14067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:12:07.370911 14061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:07.360403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50138 (local address 127.0.98.62:46527)
0924 07:12:07.360582 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:07.360594 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:07.360619 (+ 25us) server_negotiation.cc:407] Connection header received
0924 07:12:07.361080 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:07.361087 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:07.361165 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:07.361287 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:07.362048 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:07.362919 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:07.363987 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:07.364445 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:07.364566 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:07.367750 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:07.367754 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:07.367755 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:07.367766 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:07.369368 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:07.369751 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:07.369754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:07.369755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:07.369843 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:07.370056 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:07.370057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:07.370057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:07.370307 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:07.370447 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:07.370578 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 07:12:07.370713 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":90,"threads_started":1}
I20260924 07:12:07.371819 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:12:07.386580 14067 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:12:07.387065 14067 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:12:07.397568 14067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e85c9a5745e34cb7b7d89d666a9087d6: Generated new TSK 0
I20260924 07:12:07.397753 14067 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:12:07.425935 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 13990
2026-09-24T07:12:07Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1336 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:12:07 dist-test-slave-14ww krb5kdc[14085](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:12:07 dist-test-slave-14ww krb5kdc[14085](info): set up 2 sockets
Sep 24 07:12:07 dist-test-slave-14ww krb5kdc[14085](info): commencing operation
krb5kdc: starting...
W20260924 07:12:08.460798 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:12:08 dist-test-slave-14ww krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:12:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:12:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:12:08.580592 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35959
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35959
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:12:08.662249 14098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:08.662487 14098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:08.662523 14098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:08.663641 14098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:12:08.663708 14098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:08.663735 14098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:08.663765 14098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:12:08.663790 14098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:12:08.665295 14098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:12:08.665561 14098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:08.665779 14098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:08.670480 14103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:08.670616 14098 server_base.cc:1034] running on GCE node
W20260924 07:12:08.670490 14106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:08.670512 14104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:08.671017 14098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:08.671278 14098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:08.672456 14098 hybrid_clock.cc:648] HybridClock initialized: now 1790233928672455 us; error 37 us; skew 500 ppm
Sep 24 07:12:08 dist-test-slave-14ww krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:12:08.674897 14098 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:12:08.675374 14098 webserver.cc:460] Webserver started at http://127.0.98.62:37845/ using document root <none> and password file <none>
I20260924 07:12:08.675589 14098 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:08.675655 14098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:08.675814 14098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:12:08.676887 14098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "cffb68f1de7c4b2ea88bfe26f8704d8d"
format_stamp: "Formatted at 2026-09-24 07:12:08 on dist-test-slave-14ww"
server_key: "643c51f84e59fa453ea85241174da900"
server_key_iv: "712541acb897209945609f4eddf6e58b"
server_key_version: "encryptionkey@0"
I20260924 07:12:08.677246 14098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "cffb68f1de7c4b2ea88bfe26f8704d8d"
format_stamp: "Formatted at 2026-09-24 07:12:08 on dist-test-slave-14ww"
server_key: "643c51f84e59fa453ea85241174da900"
server_key_iv: "712541acb897209945609f4eddf6e58b"
server_key_version: "encryptionkey@0"
I20260924 07:12:08.678508 14098 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:12:08.679217 14113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:08.679382 14098 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:12:08.679479 14098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "cffb68f1de7c4b2ea88bfe26f8704d8d"
format_stamp: "Formatted at 2026-09-24 07:12:08 on dist-test-slave-14ww"
server_key: "643c51f84e59fa453ea85241174da900"
server_key_iv: "712541acb897209945609f4eddf6e58b"
server_key_version: "encryptionkey@0"
I20260924 07:12:08.679560 14098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:08.723135 14098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:08.725319 14098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:08.725487 14098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:08.729692 14098 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35959
I20260924 07:12:08.729768 14165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35959 every 8 connection(s)
I20260924 07:12:08.730119 14098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:12:08.731067 14166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:12:08.733294 14166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d: Bootstrap starting.
I20260924 07:12:08.733848 14166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d: Neither blocks nor log segments found. Creating new log.
I20260924 07:12:08.734122 14166 log.cc:824] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d: Log is configured to *not* fsync() on all Append() calls
I20260924 07:12:08.734771 14166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d: No bootstrap required, opened a new log
I20260924 07:12:08.735569 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 14098
I20260924 07:12:08.735653 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:12:08.735839 392 external_mini_cluster.cc:1428] Setting key 4e167bd26473d06f1482786b3d67832a
I20260924 07:12:08.736549 14166 raft_consensus.cc:348] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cffb68f1de7c4b2ea88bfe26f8704d8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35959 } }
I20260924 07:12:08.736704 14166 raft_consensus.cc:374] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:12:08.736771 14166 raft_consensus.cc:729] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cffb68f1de7c4b2ea88bfe26f8704d8d, State: Initialized, Role: FOLLOWER
I20260924 07:12:08.736917 14166 consensus_queue.cc:260] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cffb68f1de7c4b2ea88bfe26f8704d8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35959 } }
I20260924 07:12:08.737008 14166 raft_consensus.cc:388] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:12:08.737066 14166 raft_consensus.cc:482] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:12:08.737119 14166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:12:08.737926 14166 raft_consensus.cc:504] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cffb68f1de7c4b2ea88bfe26f8704d8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35959 } }
I20260924 07:12:08.738041 14166 leader_election.cc:302] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cffb68f1de7c4b2ea88bfe26f8704d8d; no voters:
I20260924 07:12:08.738214 14166 leader_election.cc:288] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:12:08.738242 14171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:12:08.738344 14171 raft_consensus.cc:686] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [term 1 LEADER]: Becoming Leader. State: Replica: cffb68f1de7c4b2ea88bfe26f8704d8d, State: Running, Role: LEADER
I20260924 07:12:08.738456 14166 sys_catalog.cc:564] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:12:08.738480 14171 consensus_queue.cc:237] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cffb68f1de7c4b2ea88bfe26f8704d8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35959 } }
I20260924 07:12:08.738889 14172 sys_catalog.cc:455] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cffb68f1de7c4b2ea88bfe26f8704d8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cffb68f1de7c4b2ea88bfe26f8704d8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35959 } } }
I20260924 07:12:08.738976 14172 sys_catalog.cc:458] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [sys.catalog]: This master's current role is: LEADER
I20260924 07:12:08.739207 14173 sys_catalog.cc:455] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader cffb68f1de7c4b2ea88bfe26f8704d8d. Latest consensus state: current_term: 1 leader_uuid: "cffb68f1de7c4b2ea88bfe26f8704d8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cffb68f1de7c4b2ea88bfe26f8704d8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35959 } } }
I20260924 07:12:08.739537 14173 sys_catalog.cc:458] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [sys.catalog]: This master's current role is: LEADER
I20260924 07:12:08.739691 14178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:12:08.740262 14178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:12:08.742094 14178 catalog_manager.cc:1347] Generated new cluster ID: 140a175d6f164024ac21a3e82d632081
I20260924 07:12:08.742148 14178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 07:12:08 dist-test-slave-14ww krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:12:08.747100 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:08.736378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54642 (local address 127.0.98.62:35959)
0924 07:12:08.736564 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:08.736572 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:08.736598 (+ 26us) server_negotiation.cc:407] Connection header received
0924 07:12:08.736686 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:08.736691 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:08.736812 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:08.736960 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:08.737611 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:08.738735 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:08.739789 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:08.740497 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:08.740611 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:08.743252 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:08.743256 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:08.743257 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:08.743268 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:08.745316 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:08.745691 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:08.745695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:08.745695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:08.745826 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:08.746053 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:08.746054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:08.746054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:08.746372 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:08.746527 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:08.746624 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 07:12:08.746760 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
I20260924 07:12:08.764271 14178 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:12:08.765041 14178 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:12:08.768574 14178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d: Generated new TSK 0
I20260924 07:12:08.768762 14178 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 07:12:08.800180 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35959
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:12:08.880443 14192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:08.880646 14192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:08.880669 14192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:08.881846 14192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:08.881891 14192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:08.881916 14192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:12:08.883673 14192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:12:08.884132 14192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:08.884399 14192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:08.889364 14200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:08.889390 14198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:08.889396 14197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:08.889498 14192 server_base.cc:1034] running on GCE node
I20260924 07:12:08.889784 14192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:08.890049 14192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:08.891232 14192 hybrid_clock.cc:648] HybridClock initialized: now 1790233928891217 us; error 51 us; skew 500 ppm
Sep 24 07:12:08 dist-test-slave-14ww krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:12:08.893874 14192 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:12:08.894307 14192 webserver.cc:460] Webserver started at http://127.0.98.1:41621/ using document root <none> and password file <none>
I20260924 07:12:08.894539 14192 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:08.894609 14192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:08.894760 14192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:12:08.895851 14192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/data/instance:
uuid: "26814bd24cdf410b9d59f11070bd7de3"
format_stamp: "Formatted at 2026-09-24 07:12:08 on dist-test-slave-14ww"
server_key: "917fea4663865ca7fdb4ab9031b87b75"
server_key_iv: "c3c358de4a1f2198501bb77d2de9c4a2"
server_key_version: "encryptionkey@0"
I20260924 07:12:08.896214 14192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/wal/instance:
uuid: "26814bd24cdf410b9d59f11070bd7de3"
format_stamp: "Formatted at 2026-09-24 07:12:08 on dist-test-slave-14ww"
server_key: "917fea4663865ca7fdb4ab9031b87b75"
server_key_iv: "c3c358de4a1f2198501bb77d2de9c4a2"
server_key_version: "encryptionkey@0"
I20260924 07:12:08.897576 14192 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:12:08.898294 14207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:08.898468 14192 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:08.898571 14192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/wal
uuid: "26814bd24cdf410b9d59f11070bd7de3"
format_stamp: "Formatted at 2026-09-24 07:12:08 on dist-test-slave-14ww"
server_key: "917fea4663865ca7fdb4ab9031b87b75"
server_key_iv: "c3c358de4a1f2198501bb77d2de9c4a2"
server_key_version: "encryptionkey@0"
I20260924 07:12:08.898649 14192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:08.907593 14192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:08.909451 14192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:08.909610 14192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:08.909806 14192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:12:08.910113 14192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:12:08.910169 14192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:08.910211 14192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:12:08.910239 14192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:08.915891 14192 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33485
I20260924 07:12:08.915934 14320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33485 every 8 connection(s)
I20260924 07:12:08.916229 14192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 07:12:08 dist-test-slave-14ww krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:12:08.924703 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 14192
I20260924 07:12:08.924834 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:12:08.925045 392 external_mini_cluster.cc:1428] Setting key bb55c06c49ac768dd79e81ba1b92515f
I20260924 07:12:08.925261 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:08.916911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40101 (local address 127.0.98.62:35959)
0924 07:12:08.916984 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:08.916987 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:08.917187 (+ 200us) server_negotiation.cc:407] Connection header received
0924 07:12:08.917644 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:08.917645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:08.917658 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:08.917708 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:08.918572 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:08.919070 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:08.920078 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:08.920265 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:08.920313 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:08.922400 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:08.922402 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:08.922402 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:08.922411 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:08.923719 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:08.924130 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:08.924131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:08.924132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:08.924191 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:08.924433 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:08.924434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:08.924434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:08.924586 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:08.924679 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:08.925070 (+ 391us) server_negotiation.cc:299] Negotiation successful
0924 07:12:08.925161 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:12:08.925486 14323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:08.916916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.1:40101)
0924 07:12:08.917064 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:08.917098 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:08.917555 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:08.917760 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:08.917763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:08.918308 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:08.918494 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:08.918499 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:08.919160 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:08.919161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:08.920003 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:08.920006 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:08.920335 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:08.920336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:08.920429 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:08.920947 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:08.920955 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:08.922300 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:12:08.923815 (+ 1515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:08.923821 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:08.923823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:08.924038 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:08.924260 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:08.924261 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:08.924261 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:08.924364 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:08.924669 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:12:08.924675 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:12:08.924879 (+ 204us) client_negotiation.cc:769] Sending connection context
0924 07:12:08.925043 (+ 164us) client_negotiation.cc:240] Negotiation successful
0924 07:12:08.925230 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
I20260924 07:12:08.925966 14321 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35959
I20260924 07:12:08.926065 14321 heartbeater.cc:464] Registering TS with master...
I20260924 07:12:08.926251 14321 heartbeater.cc:511] Master 127.0.98.62:35959 requested a full tablet report, sending...
I20260924 07:12:08.926780 14130 ts_manager.cc:194] Registered new tserver with Master: 26814bd24cdf410b9d59f11070bd7de3 (127.0.98.1:33485)
I20260924 07:12:08.927716 14130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40101
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 07:12:08.951823 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35959
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:12:09.030946 14326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:09.031176 14326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:09.031220 14326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:09.032810 14326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:09.032852 14326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:09.032879 14326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:12:09.034212 14326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:12:09.034376 14326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:09.034534 14326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:09.039368 14332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:09.039377 14334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:09.039453 14331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:09.039666 14326 server_base.cc:1034] running on GCE node
I20260924 07:12:09.039848 14326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:09.040038 14326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:09.041173 14326 hybrid_clock.cc:648] HybridClock initialized: now 1790233929041163 us; error 26 us; skew 500 ppm
Sep 24 07:12:09 dist-test-slave-14ww krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:12:09.052572 14326 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:12:09.053056 14326 webserver.cc:460] Webserver started at http://127.0.98.2:34881/ using document root <none> and password file <none>
I20260924 07:12:09.053234 14326 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:09.053273 14326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:09.053357 14326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:12:09.054332 14326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/data/instance:
uuid: "b0791f48efd344f580083d4d6eb70607"
format_stamp: "Formatted at 2026-09-24 07:12:09 on dist-test-slave-14ww"
server_key: "e86fd48a9bcd68d8f3bc8db61f62579d"
server_key_iv: "f2245b2c847cf066aed299f5d3b208c7"
server_key_version: "encryptionkey@0"
I20260924 07:12:09.054629 14326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b0791f48efd344f580083d4d6eb70607"
format_stamp: "Formatted at 2026-09-24 07:12:09 on dist-test-slave-14ww"
server_key: "e86fd48a9bcd68d8f3bc8db61f62579d"
server_key_iv: "f2245b2c847cf066aed299f5d3b208c7"
server_key_version: "encryptionkey@0"
I20260924 07:12:09.055819 14326 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:12:09.056658 14341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:09.056849 14326 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:12:09.056962 14326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/wal
uuid: "b0791f48efd344f580083d4d6eb70607"
format_stamp: "Formatted at 2026-09-24 07:12:09 on dist-test-slave-14ww"
server_key: "e86fd48a9bcd68d8f3bc8db61f62579d"
server_key_iv: "f2245b2c847cf066aed299f5d3b208c7"
server_key_version: "encryptionkey@0"
I20260924 07:12:09.057065 14326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:09.075361 14326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:09.077277 14326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:09.077433 14326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:09.077661 14326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:12:09.077955 14326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:12:09.078009 14326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:09.078048 14326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:12:09.078073 14326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:09.083901 14326 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42749
I20260924 07:12:09.083948 14454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42749 every 8 connection(s)
I20260924 07:12:09.084276 14326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:12:09.086455 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 14326
I20260924 07:12:09.086573 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:12:09.086789 392 external_mini_cluster.cc:1428] Setting key c245fea0b1e742f2d996a79c35487db7
Sep 24 07:12:09 dist-test-slave-14ww krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 07:12:09.094434 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:09.085050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32915 (local address 127.0.98.62:35959)
0924 07:12:09.085097 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:09.085101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:09.085232 (+ 131us) server_negotiation.cc:407] Connection header received
0924 07:12:09.085716 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:09.085717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:09.085727 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:09.085824 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:09.086929 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.087612 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.089168 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.089372 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.089459 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:09.091527 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:09.091528 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:09.091529 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:09.091536 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:09.093037 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:09.093469 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:09.093470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:09.093471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:09.093529 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:09.093781 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:09.093782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:09.093783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:09.093898 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:09.093977 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:09.094248 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 07:12:09.094342 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 07:12:09.094597 14457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:09.084967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.2:32915)
0924 07:12:09.085113 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:09.085162 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:09.085633 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:09.085871 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:09.085874 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:09.086551 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:09.086831 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:09.086838 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.087703 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.087705 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:09.089085 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:09.089089 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.089468 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.089469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:09.089602 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:09.090116 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:09.090130 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:09.091429 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:12:09.093156 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:09.093162 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:09.093165 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:09.093376 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:09.093605 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:09.093606 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:09.093606 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:09.093713 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:09.093966 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:12:09.093972 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:12:09.094109 (+ 137us) client_negotiation.cc:769] Sending connection context
0924 07:12:09.094216 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 07:12:09.094368 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260924 07:12:09.095086 14455 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35959
I20260924 07:12:09.095201 14455 heartbeater.cc:464] Registering TS with master...
I20260924 07:12:09.095427 14455 heartbeater.cc:511] Master 127.0.98.62:35959 requested a full tablet report, sending...
I20260924 07:12:09.095818 14130 ts_manager.cc:194] Registered new tserver with Master: b0791f48efd344f580083d4d6eb70607 (127.0.98.2:42749)
I20260924 07:12:09.096223 14130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:32915
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 07:12:09.112257 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35959
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:12:09.192015 14460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:09.192203 14460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:09.192226 14460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:09.193295 14460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:09.193328 14460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:09.193352 14460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:12:09.194698 14460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:12:09.194871 14460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:09.195034 14460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:12:09.200101 14460 server_base.cc:1034] running on GCE node
W20260924 07:12:09.200065 14468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:09.200095 14465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:09.200048 14466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:09.200572 14460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:09.200836 14460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:09.201985 14460 hybrid_clock.cc:648] HybridClock initialized: now 1790233929201955 us; error 47 us; skew 500 ppm
Sep 24 07:12:09 dist-test-slave-14ww krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:12:09.204485 14460 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:12:09.204957 14460 webserver.cc:460] Webserver started at http://127.0.98.3:42137/ using document root <none> and password file <none>
I20260924 07:12:09.205144 14460 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:09.205188 14460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:09.205272 14460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:12:09.206235 14460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/data/instance:
uuid: "41c1d10160a245ccb85f17a8403be416"
format_stamp: "Formatted at 2026-09-24 07:12:09 on dist-test-slave-14ww"
server_key: "16a3564e3c8b7279ef844e12b0181f0d"
server_key_iv: "bb7d5fdee6d3917e055244b76b686df6"
server_key_version: "encryptionkey@0"
I20260924 07:12:09.206543 14460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/wal/instance:
uuid: "41c1d10160a245ccb85f17a8403be416"
format_stamp: "Formatted at 2026-09-24 07:12:09 on dist-test-slave-14ww"
server_key: "16a3564e3c8b7279ef844e12b0181f0d"
server_key_iv: "bb7d5fdee6d3917e055244b76b686df6"
server_key_version: "encryptionkey@0"
I20260924 07:12:09.207770 14460 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:12:09.208546 14475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:09.208722 14460 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:12:09.208866 14460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/wal
uuid: "41c1d10160a245ccb85f17a8403be416"
format_stamp: "Formatted at 2026-09-24 07:12:09 on dist-test-slave-14ww"
server_key: "16a3564e3c8b7279ef844e12b0181f0d"
server_key_iv: "bb7d5fdee6d3917e055244b76b686df6"
server_key_version: "encryptionkey@0"
I20260924 07:12:09.208956 14460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:09.221791 14460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:09.223618 14460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:09.223766 14460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:09.223965 14460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:12:09.224257 14460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:12:09.224311 14460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:09.224349 14460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:12:09.224377 14460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:09.230433 14460 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40703
I20260924 07:12:09.230485 14588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40703 every 8 connection(s)
I20260924 07:12:09.230780 14460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:12:09.236833 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 14460
I20260924 07:12:09.236948 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:12:09.237149 392 external_mini_cluster.cc:1428] Setting key 3c897c6416a15853c5ae64389a323527
Sep 24 07:12:09 dist-test-slave-14ww krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:12:09.240901 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:09.231756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49773 (local address 127.0.98.62:35959)
0924 07:12:09.231803 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:09.231805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:09.232781 (+ 976us) server_negotiation.cc:407] Connection header received
0924 07:12:09.233225 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:09.233226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:09.233234 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:09.233284 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:09.234119 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.234591 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.235562 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.235745 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.235796 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:09.238145 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:09.238146 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:09.238147 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:09.238154 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:09.239702 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:09.240140 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:09.240141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:09.240141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:09.240191 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:09.240386 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:09.240386 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:09.240386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:09.240473 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:09.240522 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:09.240784 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 07:12:09.240841 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 07:12:09.241062 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:09.231428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.3:49773)
0924 07:12:09.232640 (+ 1212us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:09.232682 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:09.233143 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:09.233333 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:09.233336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:09.233863 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:09.234046 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:09.234050 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.234673 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.234674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:09.235485 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:09.235487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.235821 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.235822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:09.235946 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:09.236372 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:09.236380 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:09.238016 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:12:09.239777 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:09.239782 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:09.239784 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:09.240072 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:09.240248 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:09.240249 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:09.240249 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:09.240339 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:09.240526 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:12:09.240529 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:12:09.240640 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 07:12:09.240717 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 07:12:09.240884 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1171,"thread_start_us":111,"threads_started":1}
I20260924 07:12:09.241536 14589 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35959
I20260924 07:12:09.241609 14589 heartbeater.cc:464] Registering TS with master...
I20260924 07:12:09.241772 14589 heartbeater.cc:511] Master 127.0.98.62:35959 requested a full tablet report, sending...
I20260924 07:12:09.242116 14130 ts_manager.cc:194] Registered new tserver with Master: 41c1d10160a245ccb85f17a8403be416 (127.0.98.3:40703)
I20260924 07:12:09.242431 14130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49773
I20260924 07:12:09.248848 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:12:09.255362 14170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:09.250064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54644 (local address 127.0.98.62:35959)
0924 07:12:09.250152 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:09.250156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:09.250198 (+ 42us) server_negotiation.cc:407] Connection header received
0924 07:12:09.250235 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:09.250236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:09.250243 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:09.250289 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:09.250783 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.251226 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.251876 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.252039 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.252085 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:09.253136 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:09.253137 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:09.253138 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:09.253143 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:09.254541 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:09.254818 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:09.254819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:09.254819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:09.254867 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:09.254981 (+ 114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:09.254981 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:09.254982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:09.255086 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:09.255169 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:09.255237 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 07:12:09.255286 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 07:12:09.256841 14130 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:12:09.258316 14130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:12:09.264137 14598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:09.260322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42749 (local address 127.0.0.1:56562)
0924 07:12:09.260634 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:09.260652 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:09.260716 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:09.260973 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:09.260975 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:09.260976 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:12:09.261031 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:09.261032 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.261972 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.261973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:09.262886 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:09.262888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.263856 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.263857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:09.263931 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:09.263943 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 07:12:09.264020 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 07:12:09.264070 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":61,"threads_started":1}
I20260924 07:12:09.264153 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:09.260484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56562 (local address 127.0.98.2:42749)
0924 07:12:09.260824 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:09.260827 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:09.260836 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:12:09.260877 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:09.260879 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:09.260890 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:09.260956 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:12:09.261110 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.261870 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.263046 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.263778 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.263856 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:09.263913 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:09.264033 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 07:12:09.264075 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":62,"threads_started":1}
I20260924 07:12:09.264854 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:09.260282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33485 (local address 127.0.0.1:60562)
0924 07:12:09.260612 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:09.260652 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:09.260716 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:09.261287 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:09.261288 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:09.261290 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:12:09.261329 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:09.261331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.262606 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.262607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:09.263635 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:09.263638 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.264587 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.264588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:09.264649 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:09.264654 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:12:09.264711 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 07:12:09.264772 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":91,"threads_started":1}
I20260924 07:12:09.264879 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:09.260415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60562 (local address 127.0.98.1:33485)
0924 07:12:09.261059 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:09.261063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:09.261072 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:12:09.261111 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:09.261113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:09.261136 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:09.261206 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:12:09.261388 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.262498 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.263724 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.264437 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.264504 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:09.264553 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:09.264759 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 07:12:09.264803 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":97,"threads_started":1}
I20260924 07:12:09.265002 14389 tablet_service.cc:1467] Processing CreateTablet for tablet 46d1ece1ec0c46b881bcaa6a24933ac0 (DEFAULT_TABLE table=test-table [id=8a5899b59e1641b182cab1f4f456c45f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:12:09.265204 14601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:09.260523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40703 (local address 127.0.0.1:38168)
0924 07:12:09.261416 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:09.261427 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:09.261469 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:09.261700 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:09.261701 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:09.261703 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:12:09.261741 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:09.261742 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.262855 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.262856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:09.263795 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:09.263798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.264910 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.264911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:09.265038 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:09.265041 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:12:09.265100 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 07:12:09.265138 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"thread_start_us":53,"threads_started":1}
I20260924 07:12:09.265252 14602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:09.260591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38168 (local address 127.0.98.3:40703)
0924 07:12:09.260766 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:09.260772 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:09.261523 (+ 751us) server_negotiation.cc:407] Connection header received
0924 07:12:09.261558 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:09.261560 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:09.261576 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:09.261639 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:12:09.261808 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.262758 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.263884 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.264774 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.264830 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:09.264877 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:09.265136 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 07:12:09.265173 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
I20260924 07:12:09.265355 14389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d1ece1ec0c46b881bcaa6a24933ac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:12:09.265769 14523 tablet_service.cc:1467] Processing CreateTablet for tablet 46d1ece1ec0c46b881bcaa6a24933ac0 (DEFAULT_TABLE table=test-table [id=8a5899b59e1641b182cab1f4f456c45f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:12:09.265769 14255 tablet_service.cc:1467] Processing CreateTablet for tablet 46d1ece1ec0c46b881bcaa6a24933ac0 (DEFAULT_TABLE table=test-table [id=8a5899b59e1641b182cab1f4f456c45f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:12:09.266059 14255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d1ece1ec0c46b881bcaa6a24933ac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:12:09.266059 14523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d1ece1ec0c46b881bcaa6a24933ac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:12:09.268007 14603 tablet_bootstrap.cc:492] T 46d1ece1ec0c46b881bcaa6a24933ac0 P b0791f48efd344f580083d4d6eb70607: Bootstrap starting.
I20260924 07:12:09.268673 14603 tablet_bootstrap.cc:654] T 46d1ece1ec0c46b881bcaa6a24933ac0 P b0791f48efd344f580083d4d6eb70607: Neither blocks nor log segments found. Creating new log.
I20260924 07:12:09.269129 14604 tablet_bootstrap.cc:492] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 26814bd24cdf410b9d59f11070bd7de3: Bootstrap starting.
I20260924 07:12:09.269289 14603 log.cc:824] T 46d1ece1ec0c46b881bcaa6a24933ac0 P b0791f48efd344f580083d4d6eb70607: Log is configured to *not* fsync() on all Append() calls
I20260924 07:12:09.269316 14605 tablet_bootstrap.cc:492] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 41c1d10160a245ccb85f17a8403be416: Bootstrap starting.
I20260924 07:12:09.269816 14604 tablet_bootstrap.cc:654] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 26814bd24cdf410b9d59f11070bd7de3: Neither blocks nor log segments found. Creating new log.
I20260924 07:12:09.269989 14605 tablet_bootstrap.cc:654] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 41c1d10160a245ccb85f17a8403be416: Neither blocks nor log segments found. Creating new log.
I20260924 07:12:09.270148 14603 tablet_bootstrap.cc:492] T 46d1ece1ec0c46b881bcaa6a24933ac0 P b0791f48efd344f580083d4d6eb70607: No bootstrap required, opened a new log
I20260924 07:12:09.270231 14603 ts_tablet_manager.cc:1397] T 46d1ece1ec0c46b881bcaa6a24933ac0 P b0791f48efd344f580083d4d6eb70607: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:12:09.270372 14604 log.cc:824] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 26814bd24cdf410b9d59f11070bd7de3: Log is configured to *not* fsync() on all Append() calls
I20260924 07:12:09.270541 14605 log.cc:824] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 41c1d10160a245ccb85f17a8403be416: Log is configured to *not* fsync() on all Append() calls
I20260924 07:12:09.271188 14604 tablet_bootstrap.cc:492] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 26814bd24cdf410b9d59f11070bd7de3: No bootstrap required, opened a new log
I20260924 07:12:09.271270 14604 ts_tablet_manager.cc:1397] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 26814bd24cdf410b9d59f11070bd7de3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:12:09.271417 14605 tablet_bootstrap.cc:492] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 41c1d10160a245ccb85f17a8403be416: No bootstrap required, opened a new log
I20260924 07:12:09.271485 14605 ts_tablet_manager.cc:1397] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 41c1d10160a245ccb85f17a8403be416: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:12:09.272325 14603 raft_consensus.cc:348] T 46d1ece1ec0c46b881bcaa6a24933ac0 P b0791f48efd344f580083d4d6eb70607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26814bd24cdf410b9d59f11070bd7de3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33485 } } peers { permanent_uuid: "b0791f48efd344f580083d4d6eb70607" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 } } peers { permanent_uuid: "41c1d10160a245ccb85f17a8403be416" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 } }
I20260924 07:12:09.272508 14603 raft_consensus.cc:374] T 46d1ece1ec0c46b881bcaa6a24933ac0 P b0791f48efd344f580083d4d6eb70607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:12:09.272547 14603 raft_consensus.cc:729] T 46d1ece1ec0c46b881bcaa6a24933ac0 P b0791f48efd344f580083d4d6eb70607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0791f48efd344f580083d4d6eb70607, State: Initialized, Role: FOLLOWER
I20260924 07:12:09.272682 14603 consensus_queue.cc:260] T 46d1ece1ec0c46b881bcaa6a24933ac0 P b0791f48efd344f580083d4d6eb70607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26814bd24cdf410b9d59f11070bd7de3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33485 } } peers { permanent_uuid: "b0791f48efd344f580083d4d6eb70607" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 } } peers { permanent_uuid: "41c1d10160a245ccb85f17a8403be416" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 } }
I20260924 07:12:09.272822 14604 raft_consensus.cc:348] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 26814bd24cdf410b9d59f11070bd7de3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26814bd24cdf410b9d59f11070bd7de3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33485 } } peers { permanent_uuid: "b0791f48efd344f580083d4d6eb70607" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 } } peers { permanent_uuid: "41c1d10160a245ccb85f17a8403be416" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 } }
I20260924 07:12:09.272982 14455 heartbeater.cc:503] Master 127.0.98.62:35959 was elected leader, sending a full tablet report...
I20260924 07:12:09.272993 14604 raft_consensus.cc:374] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 26814bd24cdf410b9d59f11070bd7de3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:12:09.273041 14604 raft_consensus.cc:729] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 26814bd24cdf410b9d59f11070bd7de3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26814bd24cdf410b9d59f11070bd7de3, State: Initialized, Role: FOLLOWER
I20260924 07:12:09.273155 14604 consensus_queue.cc:260] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 26814bd24cdf410b9d59f11070bd7de3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26814bd24cdf410b9d59f11070bd7de3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33485 } } peers { permanent_uuid: "b0791f48efd344f580083d4d6eb70607" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 } } peers { permanent_uuid: "41c1d10160a245ccb85f17a8403be416" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 } }
I20260924 07:12:09.273242 14603 ts_tablet_manager.cc:1428] T 46d1ece1ec0c46b881bcaa6a24933ac0 P b0791f48efd344f580083d4d6eb70607: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 07:12:09.273411 14604 ts_tablet_manager.cc:1428] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 26814bd24cdf410b9d59f11070bd7de3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:12:09.273447 14321 heartbeater.cc:503] Master 127.0.98.62:35959 was elected leader, sending a full tablet report...
I20260924 07:12:09.273929 14605 raft_consensus.cc:348] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 41c1d10160a245ccb85f17a8403be416 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26814bd24cdf410b9d59f11070bd7de3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33485 } } peers { permanent_uuid: "b0791f48efd344f580083d4d6eb70607" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 } } peers { permanent_uuid: "41c1d10160a245ccb85f17a8403be416" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 } }
I20260924 07:12:09.274096 14605 raft_consensus.cc:374] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 41c1d10160a245ccb85f17a8403be416 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:12:09.274132 14605 raft_consensus.cc:729] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 41c1d10160a245ccb85f17a8403be416 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c1d10160a245ccb85f17a8403be416, State: Initialized, Role: FOLLOWER
I20260924 07:12:09.274251 14605 consensus_queue.cc:260] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 41c1d10160a245ccb85f17a8403be416 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26814bd24cdf410b9d59f11070bd7de3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33485 } } peers { permanent_uuid: "b0791f48efd344f580083d4d6eb70607" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 } } peers { permanent_uuid: "41c1d10160a245ccb85f17a8403be416" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 } }
I20260924 07:12:09.274513 14605 ts_tablet_manager.cc:1428] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 41c1d10160a245ccb85f17a8403be416: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 07:12:09.274525 14589 heartbeater.cc:503] Master 127.0.98.62:35959 was elected leader, sending a full tablet report...
W20260924 07:12:09.334905 14456 tablet.cc:2369] T 46d1ece1ec0c46b881bcaa6a24933ac0 P b0791f48efd344f580083d4d6eb70607: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:12:09.416913 14322 tablet.cc:2369] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 26814bd24cdf410b9d59f11070bd7de3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:12:09.433255 14609 raft_consensus.cc:482] T 46d1ece1ec0c46b881bcaa6a24933ac0 P b0791f48efd344f580083d4d6eb70607 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:12:09.433368 14609 raft_consensus.cc:504] T 46d1ece1ec0c46b881bcaa6a24933ac0 P b0791f48efd344f580083d4d6eb70607 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26814bd24cdf410b9d59f11070bd7de3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33485 } } peers { permanent_uuid: "b0791f48efd344f580083d4d6eb70607" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 } } peers { permanent_uuid: "41c1d10160a245ccb85f17a8403be416" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 } }
I20260924 07:12:09.433794 14609 leader_election.cc:288] T 46d1ece1ec0c46b881bcaa6a24933ac0 P b0791f48efd344f580083d4d6eb70607 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26814bd24cdf410b9d59f11070bd7de3 (127.0.98.1:33485), 41c1d10160a245ccb85f17a8403be416 (127.0.98.3:40703)
I20260924 07:12:09.437668 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:09.434029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58137 (local address 127.0.98.1:33485)
0924 07:12:09.434091 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:09.434094 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:09.434108 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:12:09.434161 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:09.434162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:09.434175 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:09.434232 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:12:09.434402 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.435227 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.436716 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.437315 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.437378 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:09.437385 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:09.437560 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 07:12:09.437599 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 07:12:09.437668 14602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:09.434193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54673 (local address 127.0.98.3:40703)
0924 07:12:09.434238 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:09.434239 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:09.434342 (+ 103us) server_negotiation.cc:407] Connection header received
0924 07:12:09.434383 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:09.434384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:09.434397 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:09.434450 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:12:09.434584 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.435266 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.436717 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.437294 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.437375 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:09.437385 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:09.437560 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 07:12:09.437599 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 07:12:09.437695 14457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:09.433965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33485 (local address 127.0.98.2:58137)
0924 07:12:09.434023 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:09.434036 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:09.434082 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:09.434253 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:09.434254 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:09.434256 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:12:09.434331 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:09.434333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.435334 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.435335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:09.436616 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:09.436620 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.437388 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.437390 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:09.437477 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:09.437479 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 07:12:09.437541 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 07:12:09.437590 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":39}
I20260924 07:12:09.437696 14612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:09.434085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40703 (local address 127.0.98.2:54673)
0924 07:12:09.434253 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:09.434263 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:09.434318 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:09.434462 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:09.434463 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:09.434464 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:12:09.434515 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:09.434517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.435336 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.435336 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:09.436616 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:09.436620 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.437389 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.437390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:09.437475 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:09.437479 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:12:09.437541 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 07:12:09.437589 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":81,"thread_start_us":109,"threads_started":1}
I20260924 07:12:09.438073 14275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d1ece1ec0c46b881bcaa6a24933ac0" candidate_uuid: "b0791f48efd344f580083d4d6eb70607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26814bd24cdf410b9d59f11070bd7de3" is_pre_election: true
I20260924 07:12:09.438094 14543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d1ece1ec0c46b881bcaa6a24933ac0" candidate_uuid: "b0791f48efd344f580083d4d6eb70607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c1d10160a245ccb85f17a8403be416" is_pre_election: true
I20260924 07:12:09.438205 14275 raft_consensus.cc:2454] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 26814bd24cdf410b9d59f11070bd7de3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0791f48efd344f580083d4d6eb70607 in term 0.
I20260924 07:12:09.438244 14543 raft_consensus.cc:2454] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 41c1d10160a245ccb85f17a8403be416 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0791f48efd344f580083d4d6eb70607 in term 0.
I20260924 07:12:09.438424 14344 leader_election.cc:302] T 46d1ece1ec0c46b881bcaa6a24933ac0 P b0791f48efd344f580083d4d6eb70607 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26814bd24cdf410b9d59f11070bd7de3, b0791f48efd344f580083d4d6eb70607; no voters:
I20260924 07:12:09.438577 14609 raft_consensus.cc:2781] T 46d1ece1ec0c46b881bcaa6a24933ac0 P b0791f48efd344f580083d4d6eb70607 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:12:09.438643 14609 raft_consensus.cc:482] T 46d1ece1ec0c46b881bcaa6a24933ac0 P b0791f48efd344f580083d4d6eb70607 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:12:09.438660 14609 raft_consensus.cc:3037] T 46d1ece1ec0c46b881bcaa6a24933ac0 P b0791f48efd344f580083d4d6eb70607 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:12:09.439523 14609 raft_consensus.cc:504] T 46d1ece1ec0c46b881bcaa6a24933ac0 P b0791f48efd344f580083d4d6eb70607 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26814bd24cdf410b9d59f11070bd7de3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33485 } } peers { permanent_uuid: "b0791f48efd344f580083d4d6eb70607" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 } } peers { permanent_uuid: "41c1d10160a245ccb85f17a8403be416" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 } }
I20260924 07:12:09.439668 14609 leader_election.cc:288] T 46d1ece1ec0c46b881bcaa6a24933ac0 P b0791f48efd344f580083d4d6eb70607 [CANDIDATE]: Term 1 election: Requested vote from peers 26814bd24cdf410b9d59f11070bd7de3 (127.0.98.1:33485), 41c1d10160a245ccb85f17a8403be416 (127.0.98.3:40703)
I20260924 07:12:09.439852 14275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d1ece1ec0c46b881bcaa6a24933ac0" candidate_uuid: "b0791f48efd344f580083d4d6eb70607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26814bd24cdf410b9d59f11070bd7de3"
I20260924 07:12:09.439908 14543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d1ece1ec0c46b881bcaa6a24933ac0" candidate_uuid: "b0791f48efd344f580083d4d6eb70607" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c1d10160a245ccb85f17a8403be416"
I20260924 07:12:09.439946 14275 raft_consensus.cc:3037] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 26814bd24cdf410b9d59f11070bd7de3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:12:09.440017 14543 raft_consensus.cc:3037] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 41c1d10160a245ccb85f17a8403be416 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:12:09.440651 14275 raft_consensus.cc:2454] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 26814bd24cdf410b9d59f11070bd7de3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0791f48efd344f580083d4d6eb70607 in term 1.
I20260924 07:12:09.440794 14543 raft_consensus.cc:2454] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 41c1d10160a245ccb85f17a8403be416 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0791f48efd344f580083d4d6eb70607 in term 1.
I20260924 07:12:09.440826 14344 leader_election.cc:302] T 46d1ece1ec0c46b881bcaa6a24933ac0 P b0791f48efd344f580083d4d6eb70607 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26814bd24cdf410b9d59f11070bd7de3, b0791f48efd344f580083d4d6eb70607; no voters:
I20260924 07:12:09.440908 14609 raft_consensus.cc:2781] T 46d1ece1ec0c46b881bcaa6a24933ac0 P b0791f48efd344f580083d4d6eb70607 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:12:09.441077 14609 raft_consensus.cc:686] T 46d1ece1ec0c46b881bcaa6a24933ac0 P b0791f48efd344f580083d4d6eb70607 [term 1 LEADER]: Becoming Leader. State: Replica: b0791f48efd344f580083d4d6eb70607, State: Running, Role: LEADER
I20260924 07:12:09.441174 14609 consensus_queue.cc:237] T 46d1ece1ec0c46b881bcaa6a24933ac0 P b0791f48efd344f580083d4d6eb70607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26814bd24cdf410b9d59f11070bd7de3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33485 } } peers { permanent_uuid: "b0791f48efd344f580083d4d6eb70607" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 } } peers { permanent_uuid: "41c1d10160a245ccb85f17a8403be416" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 } }
I20260924 07:12:09.441987 14129 catalog_manager.cc:5495] T 46d1ece1ec0c46b881bcaa6a24933ac0 P b0791f48efd344f580083d4d6eb70607 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0791f48efd344f580083d4d6eb70607 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b0791f48efd344f580083d4d6eb70607" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26814bd24cdf410b9d59f11070bd7de3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0791f48efd344f580083d4d6eb70607" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41c1d10160a245ccb85f17a8403be416" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 } health_report { overall_health: UNKNOWN } } }
I20260924 07:12:09.446589 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:09.444310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56564 (local address 127.0.98.2:42749)
0924 07:12:09.444350 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:09.444357 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:09.444379 (+ 22us) server_negotiation.cc:407] Connection header received
0924 07:12:09.444450 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:09.444451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:09.444464 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:09.444519 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:12:09.444608 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.445110 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.445790 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.445963 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.446011 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:09.446241 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:12:09.446327 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:12:09.446410 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:09.446448 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 07:12:09.446465 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 07:12:09.448207 14543 raft_consensus.cc:1261] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 41c1d10160a245ccb85f17a8403be416 [term 1 FOLLOWER]: Refusing update from remote peer b0791f48efd344f580083d4d6eb70607: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:12:09.448207 14275 raft_consensus.cc:1261] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 26814bd24cdf410b9d59f11070bd7de3 [term 1 FOLLOWER]: Refusing update from remote peer b0791f48efd344f580083d4d6eb70607: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:12:09.448591 14613 consensus_queue.cc:1035] T 46d1ece1ec0c46b881bcaa6a24933ac0 P b0791f48efd344f580083d4d6eb70607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26814bd24cdf410b9d59f11070bd7de3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:12:09.448761 14609 consensus_queue.cc:1035] T 46d1ece1ec0c46b881bcaa6a24933ac0 P b0791f48efd344f580083d4d6eb70607 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c1d10160a245ccb85f17a8403be416" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:12:09.451136 14616 mvcc.cc:205] Tried to move back new op lower bound from 7332798175017263104 to 7332798174991622144. Current Snapshot: MvccSnapshot[applied={T|T < 7332798175017263104}]
I20260924 07:12:09.451215 14617 mvcc.cc:205] Tried to move back new op lower bound from 7332798175017263104 to 7332798174991622144. Current Snapshot: MvccSnapshot[applied={T|T < 7332798175017263104}]
I20260924 07:12:09.452136 14618 mvcc.cc:205] Tried to move back new op lower bound from 7332798175017263104 to 7332798174991622144. Current Snapshot: MvccSnapshot[applied={T|T < 7332798175017263104 or (T in {7332798175017263104})}]
I20260924 07:12:09.453590 14130 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:12:09.453672 14130 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:12:09.454339 14130 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d: Sending DeleteTablet for 3 replicas of tablet 46d1ece1ec0c46b881bcaa6a24933ac0
I20260924 07:12:09.454628 14255 tablet_service.cc:1514] Processing DeleteTablet for tablet 46d1ece1ec0c46b881bcaa6a24933ac0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:12:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60562
I20260924 07:12:09.454692 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 14192
I20260924 07:12:09.454710 14389 tablet_service.cc:1514] Processing DeleteTablet for tablet 46d1ece1ec0c46b881bcaa6a24933ac0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:12:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56562
I20260924 07:12:09.455226 14523 tablet_service.cc:1514] Processing DeleteTablet for tablet 46d1ece1ec0c46b881bcaa6a24933ac0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:12:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38168
I20260924 07:12:09.455439 14627 tablet_replica.cc:330] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 41c1d10160a245ccb85f17a8403be416: stopping tablet replica
I20260924 07:12:09.455541 14627 raft_consensus.cc:2229] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 41c1d10160a245ccb85f17a8403be416 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:12:09.455631 14627 raft_consensus.cc:2258] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 41c1d10160a245ccb85f17a8403be416 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:12:09.456027 14627 ts_tablet_manager.cc:1905] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 41c1d10160a245ccb85f17a8403be416: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:12:09.456876 14626 tablet_replica.cc:330] T 46d1ece1ec0c46b881bcaa6a24933ac0 P b0791f48efd344f580083d4d6eb70607: stopping tablet replica
I20260924 07:12:09.457046 14626 raft_consensus.cc:2229] T 46d1ece1ec0c46b881bcaa6a24933ac0 P b0791f48efd344f580083d4d6eb70607 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:12:09.457185 14626 raft_consensus.cc:2258] T 46d1ece1ec0c46b881bcaa6a24933ac0 P b0791f48efd344f580083d4d6eb70607 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:12:09.457659 14626 ts_tablet_manager.cc:1905] T 46d1ece1ec0c46b881bcaa6a24933ac0 P b0791f48efd344f580083d4d6eb70607: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:12:09.457985 14627 ts_tablet_manager.cc:1918] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 41c1d10160a245ccb85f17a8403be416: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:12:09.458061 14627 log.cc:1196] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 41c1d10160a245ccb85f17a8403be416: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/wal/wals/46d1ece1ec0c46b881bcaa6a24933ac0
I20260924 07:12:09.458328 14627 ts_tablet_manager.cc:1939] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 41c1d10160a245ccb85f17a8403be416: Deleting consensus metadata
I20260924 07:12:09.458855 14116 catalog_manager.cc:4841] TS 41c1d10160a245ccb85f17a8403be416 (127.0.98.3:40703): tablet 46d1ece1ec0c46b881bcaa6a24933ac0 (table test-table [id=8a5899b59e1641b182cab1f4f456c45f]) successfully deleted
I20260924 07:12:09.459527 14626 ts_tablet_manager.cc:1918] T 46d1ece1ec0c46b881bcaa6a24933ac0 P b0791f48efd344f580083d4d6eb70607: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:12:09.459650 14626 log.cc:1196] T 46d1ece1ec0c46b881bcaa6a24933ac0 P b0791f48efd344f580083d4d6eb70607: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/wal/wals/46d1ece1ec0c46b881bcaa6a24933ac0
I20260924 07:12:09.459995 14626 ts_tablet_manager.cc:1939] T 46d1ece1ec0c46b881bcaa6a24933ac0 P b0791f48efd344f580083d4d6eb70607: Deleting consensus metadata
I20260924 07:12:09.460623 14114 catalog_manager.cc:4841] TS b0791f48efd344f580083d4d6eb70607 (127.0.98.2:42749): tablet 46d1ece1ec0c46b881bcaa6a24933ac0 (table test-table [id=8a5899b59e1641b182cab1f4f456c45f]) successfully deleted
W20260924 07:12:09.461176 14116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33485 (error 108)
I20260924 07:12:09.461385 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 14326
I20260924 07:12:09.461988 14601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:09.461495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33485 (local address 127.0.0.1:60566)
0924 07:12:09.461912 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:09.461950 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396}
W20260924 07:12:09.462743 14116 catalog_manager.cc:4584] TS 26814bd24cdf410b9d59f11070bd7de3 (127.0.98.1:33485): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 46d1ece1ec0c46b881bcaa6a24933ac0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33485: connect: Connection refused (error 111)
I20260924 07:12:09.465481 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 14460
I20260924 07:12:09.469573 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 14098
I20260924 07:12:09.474291 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35959
--webserver_interface=127.0.98.62
--webserver_port=37845
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:12:09.553860 14629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:09.554090 14629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:09.554126 14629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:09.555275 14629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:12:09.555349 14629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:09.555377 14629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:09.555397 14629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:12:09.555418 14629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:12:09.556905 14629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:12:09.557168 14629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:09.557377 14629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:09.565577 14636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:09.565573 14638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:09.565661 14635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:09.565793 14629 server_base.cc:1034] running on GCE node
I20260924 07:12:09.566047 14629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:09.566340 14629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:09.567483 14629 hybrid_clock.cc:648] HybridClock initialized: now 1790233929567468 us; error 47 us; skew 500 ppm
Sep 24 07:12:09 dist-test-slave-14ww krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:12:09.569950 14629 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:12:09.570390 14629 webserver.cc:460] Webserver started at http://127.0.98.62:37845/ using document root <none> and password file <none>
I20260924 07:12:09.570626 14629 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:09.570704 14629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:09.572011 14629 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:12:09.572628 14645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:09.572813 14629 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:12:09.572921 14629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "cffb68f1de7c4b2ea88bfe26f8704d8d"
format_stamp: "Formatted at 2026-09-24 07:12:08 on dist-test-slave-14ww"
server_key: "643c51f84e59fa453ea85241174da900"
server_key_iv: "712541acb897209945609f4eddf6e58b"
server_key_version: "encryptionkey@0"
I20260924 07:12:09.573284 14629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:09.592352 14629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:09.594151 14629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:09.594316 14629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:09.598129 14629 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35959
I20260924 07:12:09.598186 14697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35959 every 8 connection(s)
I20260924 07:12:09.598567 14629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:12:09.600075 14698 sys_catalog.cc:263] Verifying existing consensus state
I20260924 07:12:09.600648 14698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d: Bootstrap starting.
I20260924 07:12:09.608543 14698 log.cc:824] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d: Log is configured to *not* fsync() on all Append() calls
I20260924 07:12:09.608819 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 14629
I20260924 07:12:09.609115 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33485
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35959
--webserver_port=41621
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:12:09.610600 14698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:12:09.610873 14698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d: Bootstrap complete.
I20260924 07:12:09.613211 14698 raft_consensus.cc:348] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cffb68f1de7c4b2ea88bfe26f8704d8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35959 } }
I20260924 07:12:09.613452 14698 raft_consensus.cc:729] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cffb68f1de7c4b2ea88bfe26f8704d8d, State: Initialized, Role: FOLLOWER
I20260924 07:12:09.613633 14698 consensus_queue.cc:260] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cffb68f1de7c4b2ea88bfe26f8704d8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35959 } }
I20260924 07:12:09.613710 14698 raft_consensus.cc:388] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:12:09.613765 14698 raft_consensus.cc:482] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:12:09.613816 14698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [term 1 FOLLOWER]: Advancing to term 2
I20260924 07:12:09.614768 14698 raft_consensus.cc:504] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cffb68f1de7c4b2ea88bfe26f8704d8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35959 } }
I20260924 07:12:09.614883 14698 leader_election.cc:302] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cffb68f1de7c4b2ea88bfe26f8704d8d; no voters:
I20260924 07:12:09.615059 14698 leader_election.cc:288] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 07:12:09.615100 14702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [term 2 FOLLOWER]: Leader election won for term 2
I20260924 07:12:09.615257 14702 raft_consensus.cc:686] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [term 2 LEADER]: Becoming Leader. State: Replica: cffb68f1de7c4b2ea88bfe26f8704d8d, State: Running, Role: LEADER
I20260924 07:12:09.615298 14698 sys_catalog.cc:564] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:12:09.615357 14702 consensus_queue.cc:237] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cffb68f1de7c4b2ea88bfe26f8704d8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35959 } }
I20260924 07:12:09.615657 14703 sys_catalog.cc:455] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cffb68f1de7c4b2ea88bfe26f8704d8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cffb68f1de7c4b2ea88bfe26f8704d8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35959 } } }
I20260924 07:12:09.615656 14702 sys_catalog.cc:455] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader cffb68f1de7c4b2ea88bfe26f8704d8d. Latest consensus state: current_term: 2 leader_uuid: "cffb68f1de7c4b2ea88bfe26f8704d8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cffb68f1de7c4b2ea88bfe26f8704d8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35959 } } }
I20260924 07:12:09.615772 14703 sys_catalog.cc:458] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [sys.catalog]: This master's current role is: LEADER
I20260924 07:12:09.615779 14702 sys_catalog.cc:458] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [sys.catalog]: This master's current role is: LEADER
I20260924 07:12:09.615986 14710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:12:09.616918 14710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:12:09.617028 14710 catalog_manager.cc:1259] Loaded cluster ID: 140a175d6f164024ac21a3e82d632081
I20260924 07:12:09.617071 14710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:12:09.617364 14710 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 07:12:09.622529 14710 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 07:12:09.625109 14710 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:12:09.625290 14710 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d: Loaded TSK: 0
I20260924 07:12:09.625494 14710 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 07:12:09.692380 14701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:09.692567 14701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:09.692590 14701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:09.693730 14701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:09.693769 14701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:09.693795 14701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:12:09.695183 14701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41621
--tserver_master_addrs=127.0.98.62:35959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:12:09.695398 14701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:09.695644 14701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:09.700585 14725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:09.700559 14726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:09.700773 14728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:09.701285 14701 server_base.cc:1034] running on GCE node
I20260924 07:12:09.701491 14701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:09.701756 14701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:09.702939 14701 hybrid_clock.cc:648] HybridClock initialized: now 1790233929702928 us; error 39 us; skew 500 ppm
Sep 24 07:12:09 dist-test-slave-14ww krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:12:09.705862 14701 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:12:09.706411 14701 webserver.cc:460] Webserver started at http://127.0.98.1:41621/ using document root <none> and password file <none>
I20260924 07:12:09.706732 14701 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:09.706816 14701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:09.708491 14701 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:12:09.709290 14735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:09.709478 14701 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 07:12:09.709553 14701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/wal
uuid: "26814bd24cdf410b9d59f11070bd7de3"
format_stamp: "Formatted at 2026-09-24 07:12:08 on dist-test-slave-14ww"
server_key: "917fea4663865ca7fdb4ab9031b87b75"
server_key_iv: "c3c358de4a1f2198501bb77d2de9c4a2"
server_key_version: "encryptionkey@0"
I20260924 07:12:09.709909 14701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:09.718175 14701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:09.720391 14701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:09.720532 14701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:09.720819 14701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:12:09.721232 14742 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 07:12:09.723065 14701 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 07:12:09.723124 14701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 07:12:09.723170 14701 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 07:12:09.723783 14701 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 07:12:09.723838 14701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 07:12:09.723898 14742 tablet_bootstrap.cc:492] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 26814bd24cdf410b9d59f11070bd7de3: Bootstrap starting.
I20260924 07:12:09.730984 14701 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33485
I20260924 07:12:09.731345 14701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:12:09.733959 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 14701
I20260924 07:12:09.734302 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42749
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35959
--webserver_port=34881
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:12:09.734879 14742 log.cc:824] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 26814bd24cdf410b9d59f11070bd7de3: Log is configured to *not* fsync() on all Append() calls
I20260924 07:12:09.737305 14742 tablet_bootstrap.cc:492] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 26814bd24cdf410b9d59f11070bd7de3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:12:09.737622 14742 tablet_bootstrap.cc:492] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 26814bd24cdf410b9d59f11070bd7de3: Bootstrap complete.
Sep 24 07:12:09 dist-test-slave-14ww krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:12:09.738778 14742 ts_tablet_manager.cc:1397] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 26814bd24cdf410b9d59f11070bd7de3: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260924 07:12:09.739996 14742 raft_consensus.cc:348] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 26814bd24cdf410b9d59f11070bd7de3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26814bd24cdf410b9d59f11070bd7de3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33485 } } peers { permanent_uuid: "b0791f48efd344f580083d4d6eb70607" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 } } peers { permanent_uuid: "41c1d10160a245ccb85f17a8403be416" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 } }
I20260924 07:12:09.740301 14742 raft_consensus.cc:729] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 26814bd24cdf410b9d59f11070bd7de3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26814bd24cdf410b9d59f11070bd7de3, State: Initialized, Role: FOLLOWER
I20260924 07:12:09.740448 14742 consensus_queue.cc:260] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 26814bd24cdf410b9d59f11070bd7de3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26814bd24cdf410b9d59f11070bd7de3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33485 } } peers { permanent_uuid: "b0791f48efd344f580083d4d6eb70607" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 } } peers { permanent_uuid: "41c1d10160a245ccb85f17a8403be416" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 } }
I20260924 07:12:09.740716 14742 ts_tablet_manager.cc:1428] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 26814bd24cdf410b9d59f11070bd7de3: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260924 07:12:09.740850 14849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33485 every 8 connection(s)
I20260924 07:12:09.742820 14852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:09.732048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.1:46047)
0924 07:12:09.732236 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:09.732278 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:09.732722 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:09.733247 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:09.733251 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:09.733526 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:09.733707 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:09.733711 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.735398 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.735399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:09.736373 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:09.736375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.737072 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.737073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:09.737195 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:09.737745 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:09.737755 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:09.739127 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:12:09.741273 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:09.741280 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:09.741281 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:09.741492 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:09.741751 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:09.741752 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:09.741753 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:09.741858 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:09.742224 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:12:09.742230 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:12:09.742357 (+ 127us) client_negotiation.cc:769] Sending connection context
0924 07:12:09.742457 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 07:12:09.742624 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":116,"threads_started":1}
I20260924 07:12:09.742846 14853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:09.732125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46047 (local address 127.0.98.62:35959)
0924 07:12:09.732871 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:09.732881 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:09.732918 (+ 37us) server_negotiation.cc:407] Connection header received
0924 07:12:09.733030 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:09.733034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:09.733073 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:09.733208 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:09.733863 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.735273 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.736465 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.736953 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.737081 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:09.739230 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:09.739234 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:09.739234 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:09.739246 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:09.741180 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:09.741583 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:09.741586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:09.741586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:09.741682 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:09.741927 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:09.741928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:09.741928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:09.742145 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:09.742230 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:09.742520 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 07:12:09.742639 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":98,"threads_started":1}
I20260924 07:12:09.743507 14850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35959
I20260924 07:12:09.743599 14850 heartbeater.cc:464] Registering TS with master...
I20260924 07:12:09.743804 14850 heartbeater.cc:511] Master 127.0.98.62:35959 requested a full tablet report, sending...
I20260924 07:12:09.744446 14662 ts_manager.cc:194] Registered new tserver with Master: 26814bd24cdf410b9d59f11070bd7de3 (127.0.98.1:33485)
I20260924 07:12:09.745565 14662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46047
I20260924 07:12:09.746340 14850 heartbeater.cc:503] Master 127.0.98.62:35959 was elected leader, sending a full tablet report...
I20260924 07:12:09.756331 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:09.745047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33485 (local address 127.0.0.1:60576)
0924 07:12:09.752281 (+ 7234us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:09.752318 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:09.752394 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:09.752660 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:09.752662 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:09.752665 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:12:09.752723 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:09.752725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.753963 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.753964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:09.755237 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:09.755241 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.756013 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.756014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:09.756095 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:09.756107 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 07:12:09.756210 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 07:12:09.756262 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7205,"thread_start_us":99,"threads_started":1}
I20260924 07:12:09.756330 14858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:09.745265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60576 (local address 127.0.98.1:33485)
0924 07:12:09.752206 (+ 6941us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:09.752211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:09.752474 (+ 263us) server_negotiation.cc:407] Connection header received
0924 07:12:09.752511 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:09.752513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:09.752527 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:09.752593 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:12:09.752828 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.753852 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.755343 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.755895 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.756010 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:09.756066 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:09.756227 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 07:12:09.756262 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6905,"thread_start_us":59,"threads_started":1}
I20260924 07:12:09.757014 14784 tablet_service.cc:1514] Processing DeleteTablet for tablet 46d1ece1ec0c46b881bcaa6a24933ac0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:12:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60576
I20260924 07:12:09.757449 14859 tablet_replica.cc:330] stopping tablet replica
I20260924 07:12:09.757571 14859 raft_consensus.cc:2229] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 26814bd24cdf410b9d59f11070bd7de3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:12:09.757634 14859 raft_consensus.cc:2258] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 26814bd24cdf410b9d59f11070bd7de3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:12:09.757939 14859 ts_tablet_manager.cc:1905] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 26814bd24cdf410b9d59f11070bd7de3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:12:09.760053 14859 ts_tablet_manager.cc:1918] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 26814bd24cdf410b9d59f11070bd7de3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:12:09.760144 14859 log.cc:1196] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 26814bd24cdf410b9d59f11070bd7de3: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/wal/wals/46d1ece1ec0c46b881bcaa6a24933ac0
I20260924 07:12:09.760366 14859 ts_tablet_manager.cc:1939] T 46d1ece1ec0c46b881bcaa6a24933ac0 P 26814bd24cdf410b9d59f11070bd7de3: Deleting consensus metadata
I20260924 07:12:09.760885 14648 catalog_manager.cc:4841] TS 26814bd24cdf410b9d59f11070bd7de3 (127.0.98.1:33485): tablet 46d1ece1ec0c46b881bcaa6a24933ac0 (table test-table [id=8a5899b59e1641b182cab1f4f456c45f]) successfully deleted
W20260924 07:12:09.826182 14855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:09.826391 14855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:09.826414 14855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:09.827575 14855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:09.827618 14855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:09.827644 14855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:12:09.829035 14855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42749
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34881
--tserver_master_addrs=127.0.98.62:35959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:12:09.829298 14855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:09.829550 14855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:09.834693 14864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:09.834693 14865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:09.834839 14855 server_base.cc:1034] running on GCE node
W20260924 07:12:09.834689 14867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:09.835108 14855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:09.835319 14855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:09.836479 14855 hybrid_clock.cc:648] HybridClock initialized: now 1790233929836465 us; error 28 us; skew 500 ppm
Sep 24 07:12:09 dist-test-slave-14ww krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:12:09.839164 14855 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:12:09.839665 14855 webserver.cc:460] Webserver started at http://127.0.98.2:34881/ using document root <none> and password file <none>
I20260924 07:12:09.839927 14855 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:09.840021 14855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:09.841492 14855 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260924 07:12:09.842092 14874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:09.842264 14855 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:12:09.842370 14855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/wal
uuid: "b0791f48efd344f580083d4d6eb70607"
format_stamp: "Formatted at 2026-09-24 07:12:09 on dist-test-slave-14ww"
server_key: "e86fd48a9bcd68d8f3bc8db61f62579d"
server_key_iv: "f2245b2c847cf066aed299f5d3b208c7"
server_key_version: "encryptionkey@0"
I20260924 07:12:09.842742 14855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:09.860428 14855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:09.862491 14855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:09.862665 14855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:09.862901 14855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:12:09.863300 14855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:12:09.863343 14855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:09.863372 14855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:12:09.863389 14855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:09.869513 14855 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42749
I20260924 07:12:09.869556 14987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42749 every 8 connection(s)
I20260924 07:12:09.869886 14855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 07:12:09 dist-test-slave-14ww krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:12:09.877282 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 14855
I20260924 07:12:09.877566 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40703
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35959
--webserver_port=42137
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:12:09.879498 14990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:09.870627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.2:42041)
0924 07:12:09.870788 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:09.870823 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:09.871289 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:09.871489 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:09.871492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:09.872071 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:09.872261 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:09.872266 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.872927 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.872928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:09.873738 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:09.873740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.874078 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.874078 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:09.874173 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:09.874600 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:09.874607 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:09.876021 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:12:09.877919 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:09.877925 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:09.877928 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:09.878239 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:09.878485 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:09.878486 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:09.878486 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:09.878625 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:09.879002 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:12:09.879008 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:12:09.879144 (+ 136us) client_negotiation.cc:769] Sending connection context
0924 07:12:09.879228 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 07:12:09.879345 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
I20260924 07:12:09.879504 14853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:09.870608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42041 (local address 127.0.98.62:35959)
0924 07:12:09.870657 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:09.870659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:09.870909 (+ 250us) server_negotiation.cc:407] Connection header received
0924 07:12:09.871368 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:09.871369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:09.871384 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:09.871436 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:09.872342 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.872855 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.873813 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:09.874003 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:09.874056 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:09.876138 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:09.876140 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:09.876140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:09.876148 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:09.877844 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:09.878357 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:09.878358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:09.878359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:09.878413 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:09.878728 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:09.878729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:09.878729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:09.878912 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:09.879005 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:09.879278 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 07:12:09.879380 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 07:12:09.880155 14988 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35959
I20260924 07:12:09.880270 14988 heartbeater.cc:464] Registering TS with master...
I20260924 07:12:09.880446 14988 heartbeater.cc:511] Master 127.0.98.62:35959 requested a full tablet report, sending...
I20260924 07:12:09.880977 14662 ts_manager.cc:194] Registered new tserver with Master: b0791f48efd344f580083d4d6eb70607 (127.0.98.2:42749)
I20260924 07:12:09.881522 14662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42041
W20260924 07:12:09.960279 14991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:09.960469 14991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:09.960490 14991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:09.961632 14991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:09.961673 14991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:09.961699 14991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:12:09.963058 14991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40703
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42137
--tserver_master_addrs=127.0.98.62:35959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:12:09.963227 14991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:09.963394 14991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:09.968122 14996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:09.968137 14999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:09.968231 14991 server_base.cc:1034] running on GCE node
W20260924 07:12:09.968124 14997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:09.968467 14991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:09.968655 14991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:09.969790 14991 hybrid_clock.cc:648] HybridClock initialized: now 1790233929969770 us; error 34 us; skew 500 ppm
Sep 24 07:12:09 dist-test-slave-14ww krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:12:09.972088 14991 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:12:09.972471 14991 webserver.cc:460] Webserver started at http://127.0.98.3:42137/ using document root <none> and password file <none>
I20260924 07:12:09.972707 14991 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:09.972810 14991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:09.974154 14991 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:12:09.974721 15006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:09.974892 14991 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:12:09.974960 14991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/wal
uuid: "41c1d10160a245ccb85f17a8403be416"
format_stamp: "Formatted at 2026-09-24 07:12:09 on dist-test-slave-14ww"
server_key: "16a3564e3c8b7279ef844e12b0181f0d"
server_key_iv: "bb7d5fdee6d3917e055244b76b686df6"
server_key_version: "encryptionkey@0"
I20260924 07:12:09.975328 14991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:09.991415 14991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:09.993809 14991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:09.993965 14991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:09.994211 14991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:12:09.994510 14991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:12:09.994541 14991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:09.994563 14991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:12:09.994611 14991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:10.000332 14991 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40703
I20260924 07:12:10.000377 15119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40703 every 8 connection(s)
I20260924 07:12:10.000691 14991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:12:10.003129 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 14991
Sep 24 07:12:10 dist-test-slave-14ww krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:12:10.010774 14853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:10.001383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38347 (local address 127.0.98.62:35959)
0924 07:12:10.001434 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:10.001436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:10.001639 (+ 203us) server_negotiation.cc:407] Connection header received
0924 07:12:10.002096 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:10.002097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:10.002106 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:10.002176 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:10.003068 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:10.003792 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:10.004830 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:10.005075 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:10.005172 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:10.007415 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:10.007416 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:10.007416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:10.007423 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:10.009178 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:10.009632 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:10.009633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:10.009634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:10.009694 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:10.009999 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:10.010000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:10.010000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:10.010162 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:10.010226 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:10.010530 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 07:12:10.010626 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 07:12:10.010789 15124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:10.003492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54656 (local address 127.0.98.62:35959)
0924 07:12:10.003591 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:10.003593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:10.003623 (+ 30us) server_negotiation.cc:407] Connection header received
0924 07:12:10.003710 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:10.003711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:10.003721 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:10.003783 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:10.004279 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:10.004819 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:10.006274 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:10.006478 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:10.006533 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:10.007848 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:10.007849 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:10.007850 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:10.007857 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:10.009356 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:10.009924 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:10.009926 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:10.009926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:10.009983 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:10.010182 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:10.010182 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:10.010183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:10.010320 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:10.010487 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:10.010538 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 07:12:10.010687 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":82,"thread_start_us":50,"threads_started":1}
I20260924 07:12:10.010890 15122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:10.001349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.3:38347)
0924 07:12:10.001510 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:10.001551 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:10.002010 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:10.002247 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:10.002250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:10.002784 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:10.002959 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:10.002964 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:10.003872 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:10.003873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:10.004689 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:10.004691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:10.005180 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:10.005181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:10.005304 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:10.005750 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:10.005758 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:10.007317 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:12:10.009259 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:10.009265 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:10.009267 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:10.009546 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:10.009772 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:10.009772 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:10.009773 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:10.009865 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:10.010233 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:12:10.010239 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:12:10.010390 (+ 151us) client_negotiation.cc:769] Sending connection context
0924 07:12:10.010496 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 07:12:10.010663 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":90,"threads_started":1}
I20260924 07:12:10.011343 15120 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35959
I20260924 07:12:10.011446 15120 heartbeater.cc:464] Registering TS with master...
I20260924 07:12:10.011636 15120 heartbeater.cc:511] Master 127.0.98.62:35959 requested a full tablet report, sending...
I20260924 07:12:10.012017 14662 ts_manager.cc:194] Registered new tserver with Master: 41c1d10160a245ccb85f17a8403be416 (127.0.98.3:40703)
I20260924 07:12:10.012434 14662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38347
I20260924 07:12:10.022809 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:12:10.025401 14853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:10.023208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54672 (local address 127.0.98.62:35959)
0924 07:12:10.023264 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:10.023266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:10.023336 (+ 70us) server_negotiation.cc:407] Connection header received
0924 07:12:10.023382 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:10.023383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:10.023412 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:10.023471 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:12:10.023567 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:10.024021 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:10.024726 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:10.024958 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:10.025012 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:10.025167 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:12:10.025253 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:12:10.025311 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:10.025341 (+ 30us) server_negotiation.cc:299] Negotiation successful
0924 07:12:10.025354 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 07:12:10.025651 14662 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54672:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:12:10.027207 14662 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:12:10.029358 14784 tablet_service.cc:1467] Processing CreateTablet for tablet c4dc70c6285042078d1adbae705bf067 (DEFAULT_TABLE table=test-table [id=86ac5e686d1b408db42aadcd3b5494ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:12:10.029580 14784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4dc70c6285042078d1adbae705bf067. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:12:10.031188 14742 tablet_bootstrap.cc:492] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3: Bootstrap starting.
I20260924 07:12:10.031513 14742 tablet_bootstrap.cc:654] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3: Neither blocks nor log segments found. Creating new log.
I20260924 07:12:10.032145 14742 tablet_bootstrap.cc:492] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3: No bootstrap required, opened a new log
I20260924 07:12:10.032193 14742 ts_tablet_manager.cc:1397] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 07:12:10.032300 14742 raft_consensus.cc:348] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c1d10160a245ccb85f17a8403be416" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 } } peers { permanent_uuid: "b0791f48efd344f580083d4d6eb70607" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 } } peers { permanent_uuid: "26814bd24cdf410b9d59f11070bd7de3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33485 } }
I20260924 07:12:10.032383 14742 raft_consensus.cc:374] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:12:10.032435 14742 raft_consensus.cc:729] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26814bd24cdf410b9d59f11070bd7de3, State: Initialized, Role: FOLLOWER
I20260924 07:12:10.032478 14742 consensus_queue.cc:260] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c1d10160a245ccb85f17a8403be416" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 } } peers { permanent_uuid: "b0791f48efd344f580083d4d6eb70607" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 } } peers { permanent_uuid: "26814bd24cdf410b9d59f11070bd7de3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33485 } }
I20260924 07:12:10.032583 14742 ts_tablet_manager.cc:1428] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 07:12:10.033496 15127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:10.029205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42749 (local address 127.0.0.1:56578)
0924 07:12:10.029345 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:10.029357 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:10.029404 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:10.029892 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:10.029893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:10.029896 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:12:10.029939 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:10.029940 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:10.031028 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:10.031029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:10.032270 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:10.032273 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:10.033219 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:10.033220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:10.033289 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:10.033295 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 07:12:10.033352 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 07:12:10.033401 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":9,"thread_start_us":78,"threads_started":1}
I20260924 07:12:10.033515 15128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:10.029433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56578 (local address 127.0.98.2:42749)
0924 07:12:10.029709 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:10.029714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:10.029726 (+ 12us) server_negotiation.cc:407] Connection header received
0924 07:12:10.029768 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:10.029770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:10.029785 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:10.029897 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:12:10.030035 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:10.030766 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:10.032345 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:10.033071 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:10.033128 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:10.033185 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:10.033387 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 07:12:10.033432 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":69,"threads_started":1}
I20260924 07:12:10.033515 15126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:10.029184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38174 (local address 127.0.98.3:40703)
0924 07:12:10.029517 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:10.029522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:10.029531 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:12:10.029566 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:10.029568 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:10.029582 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:10.029862 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:12:10.030076 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:10.031197 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:10.032315 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:10.033055 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:10.033128 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:10.033184 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:10.033386 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 07:12:10.033432 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":96,"threads_started":1}
I20260924 07:12:10.033496 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:10.029050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40703 (local address 127.0.0.1:38174)
0924 07:12:10.029085 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:10.029096 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:10.029130 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:10.029753 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:10.029753 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:10.029755 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:12:10.029792 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:10.029794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:10.031324 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:10.031325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:10.032228 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:10.032231 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:10.033219 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:10.033220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:10.033289 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:10.033294 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:12:10.033352 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 07:12:10.033401 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19,"mutex_wait_us":7}
I20260924 07:12:10.034220 15054 tablet_service.cc:1467] Processing CreateTablet for tablet c4dc70c6285042078d1adbae705bf067 (DEFAULT_TABLE table=test-table [id=86ac5e686d1b408db42aadcd3b5494ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:12:10.034575 15054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4dc70c6285042078d1adbae705bf067. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:12:10.034576 14922 tablet_service.cc:1467] Processing CreateTablet for tablet c4dc70c6285042078d1adbae705bf067 (DEFAULT_TABLE table=test-table [id=86ac5e686d1b408db42aadcd3b5494ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:12:10.034858 14922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4dc70c6285042078d1adbae705bf067. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:12:10.038066 15130 tablet_bootstrap.cc:492] T c4dc70c6285042078d1adbae705bf067 P 41c1d10160a245ccb85f17a8403be416: Bootstrap starting.
I20260924 07:12:10.038090 15131 tablet_bootstrap.cc:492] T c4dc70c6285042078d1adbae705bf067 P b0791f48efd344f580083d4d6eb70607: Bootstrap starting.
I20260924 07:12:10.038755 15131 tablet_bootstrap.cc:654] T c4dc70c6285042078d1adbae705bf067 P b0791f48efd344f580083d4d6eb70607: Neither blocks nor log segments found. Creating new log.
I20260924 07:12:10.038755 15130 tablet_bootstrap.cc:654] T c4dc70c6285042078d1adbae705bf067 P 41c1d10160a245ccb85f17a8403be416: Neither blocks nor log segments found. Creating new log.
I20260924 07:12:10.039332 15130 log.cc:824] T c4dc70c6285042078d1adbae705bf067 P 41c1d10160a245ccb85f17a8403be416: Log is configured to *not* fsync() on all Append() calls
I20260924 07:12:10.039379 15131 log.cc:824] T c4dc70c6285042078d1adbae705bf067 P b0791f48efd344f580083d4d6eb70607: Log is configured to *not* fsync() on all Append() calls
I20260924 07:12:10.040166 15131 tablet_bootstrap.cc:492] T c4dc70c6285042078d1adbae705bf067 P b0791f48efd344f580083d4d6eb70607: No bootstrap required, opened a new log
I20260924 07:12:10.040238 15131 ts_tablet_manager.cc:1397] T c4dc70c6285042078d1adbae705bf067 P b0791f48efd344f580083d4d6eb70607: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:12:10.040297 15130 tablet_bootstrap.cc:492] T c4dc70c6285042078d1adbae705bf067 P 41c1d10160a245ccb85f17a8403be416: No bootstrap required, opened a new log
I20260924 07:12:10.040364 15130 ts_tablet_manager.cc:1397] T c4dc70c6285042078d1adbae705bf067 P 41c1d10160a245ccb85f17a8403be416: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:12:10.042251 15131 raft_consensus.cc:348] T c4dc70c6285042078d1adbae705bf067 P b0791f48efd344f580083d4d6eb70607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c1d10160a245ccb85f17a8403be416" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 } } peers { permanent_uuid: "b0791f48efd344f580083d4d6eb70607" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 } } peers { permanent_uuid: "26814bd24cdf410b9d59f11070bd7de3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33485 } }
I20260924 07:12:10.042441 15131 raft_consensus.cc:374] T c4dc70c6285042078d1adbae705bf067 P b0791f48efd344f580083d4d6eb70607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:12:10.042482 15131 raft_consensus.cc:729] T c4dc70c6285042078d1adbae705bf067 P b0791f48efd344f580083d4d6eb70607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0791f48efd344f580083d4d6eb70607, State: Initialized, Role: FOLLOWER
I20260924 07:12:10.042477 15130 raft_consensus.cc:348] T c4dc70c6285042078d1adbae705bf067 P 41c1d10160a245ccb85f17a8403be416 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c1d10160a245ccb85f17a8403be416" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 } } peers { permanent_uuid: "b0791f48efd344f580083d4d6eb70607" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 } } peers { permanent_uuid: "26814bd24cdf410b9d59f11070bd7de3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33485 } }
I20260924 07:12:10.042644 15130 raft_consensus.cc:374] T c4dc70c6285042078d1adbae705bf067 P 41c1d10160a245ccb85f17a8403be416 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:12:10.042609 15131 consensus_queue.cc:260] T c4dc70c6285042078d1adbae705bf067 P b0791f48efd344f580083d4d6eb70607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c1d10160a245ccb85f17a8403be416" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 } } peers { permanent_uuid: "b0791f48efd344f580083d4d6eb70607" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 } } peers { permanent_uuid: "26814bd24cdf410b9d59f11070bd7de3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33485 } }
I20260924 07:12:10.042677 15130 raft_consensus.cc:729] T c4dc70c6285042078d1adbae705bf067 P 41c1d10160a245ccb85f17a8403be416 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41c1d10160a245ccb85f17a8403be416, State: Initialized, Role: FOLLOWER
I20260924 07:12:10.042785 15130 consensus_queue.cc:260] T c4dc70c6285042078d1adbae705bf067 P 41c1d10160a245ccb85f17a8403be416 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c1d10160a245ccb85f17a8403be416" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 } } peers { permanent_uuid: "b0791f48efd344f580083d4d6eb70607" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 } } peers { permanent_uuid: "26814bd24cdf410b9d59f11070bd7de3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33485 } }
I20260924 07:12:10.042958 14988 heartbeater.cc:503] Master 127.0.98.62:35959 was elected leader, sending a full tablet report...
I20260924 07:12:10.043056 15130 ts_tablet_manager.cc:1428] T c4dc70c6285042078d1adbae705bf067 P 41c1d10160a245ccb85f17a8403be416: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 07:12:10.043089 15120 heartbeater.cc:503] Master 127.0.98.62:35959 was elected leader, sending a full tablet report...
I20260924 07:12:10.042893 15131 ts_tablet_manager.cc:1428] T c4dc70c6285042078d1adbae705bf067 P b0791f48efd344f580083d4d6eb70607: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260924 07:12:10.120494 14989 tablet.cc:2369] T c4dc70c6285042078d1adbae705bf067 P b0791f48efd344f580083d4d6eb70607: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:12:10.241393 14851 tablet.cc:2369] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:12:10.245869 14856 raft_consensus.cc:482] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:12:10.246070 14856 raft_consensus.cc:504] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c1d10160a245ccb85f17a8403be416" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 } } peers { permanent_uuid: "b0791f48efd344f580083d4d6eb70607" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 } } peers { permanent_uuid: "26814bd24cdf410b9d59f11070bd7de3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33485 } }
I20260924 07:12:10.246440 14856 leader_election.cc:288] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41c1d10160a245ccb85f17a8403be416 (127.0.98.3:40703), b0791f48efd344f580083d4d6eb70607 (127.0.98.2:42749)
I20260924 07:12:10.249439 15126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:10.246745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40303 (local address 127.0.98.3:40703)
0924 07:12:10.246795 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:10.246797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:10.246878 (+ 81us) server_negotiation.cc:407] Connection header received
0924 07:12:10.246949 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:10.246950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:10.246965 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:10.247048 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:12:10.247224 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:10.247684 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:10.248662 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:10.249090 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:10.249180 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:10.249190 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:10.249340 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 07:12:10.249379 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 07:12:10.249461 15136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:10.246670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40703 (local address 127.0.98.1:40303)
0924 07:12:10.246794 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:10.246805 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:10.246877 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:10.247053 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:10.247054 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:10.247056 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:12:10.247122 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:10.247124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:10.247766 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:10.247767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:10.248568 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:10.248570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:10.249180 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:10.249181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:10.249251 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:10.249256 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:12:10.249327 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 07:12:10.249368 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":29,"thread_start_us":57,"threads_started":1}
I20260924 07:12:10.249716 15128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:10.246765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34563 (local address 127.0.98.2:42749)
0924 07:12:10.246809 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:10.246810 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:10.246857 (+ 47us) server_negotiation.cc:407] Connection header received
0924 07:12:10.246953 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:10.246955 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:10.246975 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:10.247041 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:12:10.247199 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:10.247675 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:10.248685 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:10.249288 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:10.249369 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:10.249378 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:10.249608 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 07:12:10.249657 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 07:12:10.249734 15074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc70c6285042078d1adbae705bf067" candidate_uuid: "26814bd24cdf410b9d59f11070bd7de3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c1d10160a245ccb85f17a8403be416" is_pre_election: true
I20260924 07:12:10.249912 15074 raft_consensus.cc:2454] T c4dc70c6285042078d1adbae705bf067 P 41c1d10160a245ccb85f17a8403be416 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26814bd24cdf410b9d59f11070bd7de3 in term 0.
I20260924 07:12:10.250102 14738 leader_election.cc:302] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26814bd24cdf410b9d59f11070bd7de3, 41c1d10160a245ccb85f17a8403be416; no voters:
I20260924 07:12:10.250114 15137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:10.246674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42749 (local address 127.0.98.1:34563)
0924 07:12:10.246795 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:10.246806 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:10.246873 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:10.247045 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:10.247046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:10.247048 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:12:10.247132 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:10.247134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:10.247758 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:10.247759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:10.248584 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:10.248586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:10.249429 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:10.249431 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:10.249510 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:10.249515 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:12:10.249796 (+ 281us) client_negotiation.cc:240] Negotiation successful
0924 07:12:10.249835 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":50,"thread_start_us":59,"threads_started":1}
I20260924 07:12:10.250252 14856 raft_consensus.cc:2781] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:12:10.250360 14856 raft_consensus.cc:482] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:12:10.250409 14856 raft_consensus.cc:3037] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:12:10.250571 14942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc70c6285042078d1adbae705bf067" candidate_uuid: "26814bd24cdf410b9d59f11070bd7de3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0791f48efd344f580083d4d6eb70607" is_pre_election: true
I20260924 07:12:10.250681 14942 raft_consensus.cc:2454] T c4dc70c6285042078d1adbae705bf067 P b0791f48efd344f580083d4d6eb70607 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26814bd24cdf410b9d59f11070bd7de3 in term 0.
W20260924 07:12:10.251178 15121 tablet.cc:2369] T c4dc70c6285042078d1adbae705bf067 P 41c1d10160a245ccb85f17a8403be416: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:12:10.251309 14856 raft_consensus.cc:504] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c1d10160a245ccb85f17a8403be416" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 } } peers { permanent_uuid: "b0791f48efd344f580083d4d6eb70607" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 } } peers { permanent_uuid: "26814bd24cdf410b9d59f11070bd7de3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33485 } }
I20260924 07:12:10.251458 14856 leader_election.cc:288] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [CANDIDATE]: Term 1 election: Requested vote from peers 41c1d10160a245ccb85f17a8403be416 (127.0.98.3:40703), b0791f48efd344f580083d4d6eb70607 (127.0.98.2:42749)
I20260924 07:12:10.251593 15074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc70c6285042078d1adbae705bf067" candidate_uuid: "26814bd24cdf410b9d59f11070bd7de3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41c1d10160a245ccb85f17a8403be416"
I20260924 07:12:10.251672 15074 raft_consensus.cc:3037] T c4dc70c6285042078d1adbae705bf067 P 41c1d10160a245ccb85f17a8403be416 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:12:10.251649 14942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4dc70c6285042078d1adbae705bf067" candidate_uuid: "26814bd24cdf410b9d59f11070bd7de3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0791f48efd344f580083d4d6eb70607"
I20260924 07:12:10.251740 14942 raft_consensus.cc:3037] T c4dc70c6285042078d1adbae705bf067 P b0791f48efd344f580083d4d6eb70607 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:12:10.252370 15074 raft_consensus.cc:2454] T c4dc70c6285042078d1adbae705bf067 P 41c1d10160a245ccb85f17a8403be416 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26814bd24cdf410b9d59f11070bd7de3 in term 1.
I20260924 07:12:10.252521 14738 leader_election.cc:302] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26814bd24cdf410b9d59f11070bd7de3, 41c1d10160a245ccb85f17a8403be416; no voters:
I20260924 07:12:10.252599 14856 raft_consensus.cc:2781] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:12:10.252616 14942 raft_consensus.cc:2454] T c4dc70c6285042078d1adbae705bf067 P b0791f48efd344f580083d4d6eb70607 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26814bd24cdf410b9d59f11070bd7de3 in term 1.
I20260924 07:12:10.252727 14856 raft_consensus.cc:686] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [term 1 LEADER]: Becoming Leader. State: Replica: 26814bd24cdf410b9d59f11070bd7de3, State: Running, Role: LEADER
I20260924 07:12:10.252853 14856 consensus_queue.cc:237] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c1d10160a245ccb85f17a8403be416" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 } } peers { permanent_uuid: "b0791f48efd344f580083d4d6eb70607" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 } } peers { permanent_uuid: "26814bd24cdf410b9d59f11070bd7de3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33485 } }
I20260924 07:12:10.254590 14661 catalog_manager.cc:5495] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26814bd24cdf410b9d59f11070bd7de3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "26814bd24cdf410b9d59f11070bd7de3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c1d10160a245ccb85f17a8403be416" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0791f48efd344f580083d4d6eb70607" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26814bd24cdf410b9d59f11070bd7de3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33485 } health_report { overall_health: HEALTHY } } }
I20260924 07:12:10.315143 15141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:10.312574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60580 (local address 127.0.98.1:33485)
0924 07:12:10.312682 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:10.312684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:10.312692 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:12:10.312754 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:10.312755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:10.312773 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:10.312879 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:12:10.312992 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:10.313559 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:10.314363 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:10.314569 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:10.314642 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:10.314799 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:12:10.314894 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:12:10.314968 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:10.315017 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 07:12:10.315046 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":48,"threads_started":1}
I20260924 07:12:10.316699 14942 raft_consensus.cc:1261] T c4dc70c6285042078d1adbae705bf067 P b0791f48efd344f580083d4d6eb70607 [term 1 FOLLOWER]: Refusing update from remote peer 26814bd24cdf410b9d59f11070bd7de3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:12:10.316702 15074 raft_consensus.cc:1261] T c4dc70c6285042078d1adbae705bf067 P 41c1d10160a245ccb85f17a8403be416 [term 1 FOLLOWER]: Refusing update from remote peer 26814bd24cdf410b9d59f11070bd7de3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:12:10.317076 15138 consensus_queue.cc:1035] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0791f48efd344f580083d4d6eb70607" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:12:10.317229 14856 consensus_queue.cc:1035] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41c1d10160a245ccb85f17a8403be416" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:12:10.319624 15142 mvcc.cc:205] Tried to move back new op lower bound from 7332798178574479360 to 7332798178316337152. Current Snapshot: MvccSnapshot[applied={T|T < 7332798178574479360}]
I20260924 07:12:10.319770 15143 mvcc.cc:205] Tried to move back new op lower bound from 7332798178574479360 to 7332798178316337152. Current Snapshot: MvccSnapshot[applied={T|T < 7332798178574479360}]
I20260924 07:12:10.320420 15144 mvcc.cc:205] Tried to move back new op lower bound from 7332798178574479360 to 7332798178316337152. Current Snapshot: MvccSnapshot[applied={T|T < 7332798178574479360}]
I20260924 07:12:10.321758 14662 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:54672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:12:10.321858 14662 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:54672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:12:10.322561 14662 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d: Sending DeleteTablet for 3 replicas of tablet c4dc70c6285042078d1adbae705bf067
I20260924 07:12:10.322863 15054 tablet_service.cc:1514] Processing DeleteTablet for tablet c4dc70c6285042078d1adbae705bf067 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:12:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38174
I20260924 07:12:10.322894 14784 tablet_service.cc:1514] Processing DeleteTablet for tablet c4dc70c6285042078d1adbae705bf067 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:12:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60576
I20260924 07:12:10.322917 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 14701
I20260924 07:12:10.323091 14922 tablet_service.cc:1514] Processing DeleteTablet for tablet c4dc70c6285042078d1adbae705bf067 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:12:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56578
I20260924 07:12:10.323593 15154 tablet_replica.cc:330] T c4dc70c6285042078d1adbae705bf067 P b0791f48efd344f580083d4d6eb70607: stopping tablet replica
I20260924 07:12:10.323683 15154 raft_consensus.cc:2229] T c4dc70c6285042078d1adbae705bf067 P b0791f48efd344f580083d4d6eb70607 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:12:10.323779 15154 raft_consensus.cc:2258] T c4dc70c6285042078d1adbae705bf067 P b0791f48efd344f580083d4d6eb70607 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:12:10.323930 15153 tablet_replica.cc:330] T c4dc70c6285042078d1adbae705bf067 P 41c1d10160a245ccb85f17a8403be416: stopping tablet replica
I20260924 07:12:10.324013 15153 raft_consensus.cc:2229] T c4dc70c6285042078d1adbae705bf067 P 41c1d10160a245ccb85f17a8403be416 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:12:10.324085 15153 raft_consensus.cc:2258] T c4dc70c6285042078d1adbae705bf067 P 41c1d10160a245ccb85f17a8403be416 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:12:10.324448 15153 ts_tablet_manager.cc:1905] T c4dc70c6285042078d1adbae705bf067 P 41c1d10160a245ccb85f17a8403be416: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:12:10.325600 15154 ts_tablet_manager.cc:1905] T c4dc70c6285042078d1adbae705bf067 P b0791f48efd344f580083d4d6eb70607: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:12:10.326269 15153 ts_tablet_manager.cc:1918] T c4dc70c6285042078d1adbae705bf067 P 41c1d10160a245ccb85f17a8403be416: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:12:10.326368 15153 log.cc:1196] T c4dc70c6285042078d1adbae705bf067 P 41c1d10160a245ccb85f17a8403be416: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/wal/wals/c4dc70c6285042078d1adbae705bf067
I20260924 07:12:10.326649 15153 ts_tablet_manager.cc:1939] T c4dc70c6285042078d1adbae705bf067 P 41c1d10160a245ccb85f17a8403be416: Deleting consensus metadata
I20260924 07:12:10.327086 14648 catalog_manager.cc:4841] TS 41c1d10160a245ccb85f17a8403be416 (127.0.98.3:40703): tablet c4dc70c6285042078d1adbae705bf067 (table test-table [id=86ac5e686d1b408db42aadcd3b5494ab]) successfully deleted
I20260924 07:12:10.327529 15154 ts_tablet_manager.cc:1918] T c4dc70c6285042078d1adbae705bf067 P b0791f48efd344f580083d4d6eb70607: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:12:10.327603 15154 log.cc:1196] T c4dc70c6285042078d1adbae705bf067 P b0791f48efd344f580083d4d6eb70607: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/wal/wals/c4dc70c6285042078d1adbae705bf067
I20260924 07:12:10.327869 15154 ts_tablet_manager.cc:1939] T c4dc70c6285042078d1adbae705bf067 P b0791f48efd344f580083d4d6eb70607: Deleting consensus metadata
I20260924 07:12:10.328401 14646 catalog_manager.cc:4841] TS b0791f48efd344f580083d4d6eb70607 (127.0.98.2:42749): tablet c4dc70c6285042078d1adbae705bf067 (table test-table [id=86ac5e686d1b408db42aadcd3b5494ab]) successfully deleted
W20260924 07:12:10.329347 14648 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33485 (error 108)
I20260924 07:12:10.329603 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 14855
I20260924 07:12:10.330207 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:10.329717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33485 (local address 127.0.0.1:60590)
0924 07:12:10.330122 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:10.330153 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379}
W20260924 07:12:10.330610 14648 catalog_manager.cc:4584] TS 26814bd24cdf410b9d59f11070bd7de3 (127.0.98.1:33485): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c4dc70c6285042078d1adbae705bf067: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33485: connect: Connection refused (error 111)
I20260924 07:12:10.335011 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 14991
I20260924 07:12:10.339370 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 14629
I20260924 07:12:10.344605 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35959
--webserver_interface=127.0.98.62
--webserver_port=37845
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35959
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:12:10.424671 15156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:10.424919 15156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:10.424955 15156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:10.426087 15156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:12:10.426151 15156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:10.426178 15156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:10.426206 15156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:12:10.426239 15156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:12:10.427659 15156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:12:10.427922 15156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:10.428148 15156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:10.433152 15161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:10.433197 15156 server_base.cc:1034] running on GCE node
W20260924 07:12:10.433147 15164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:10.433149 15162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:10.433602 15156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:10.433915 15156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:10.435070 15156 hybrid_clock.cc:648] HybridClock initialized: now 1790233930435054 us; error 34 us; skew 500 ppm
Sep 24 07:12:10 dist-test-slave-14ww krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:12:10.437412 15156 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:12:10.437765 15156 webserver.cc:460] Webserver started at http://127.0.98.62:37845/ using document root <none> and password file <none>
I20260924 07:12:10.437996 15156 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:10.438041 15156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:10.439314 15156 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:12:10.439966 15171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:10.440155 15156 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:12:10.440243 15156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "cffb68f1de7c4b2ea88bfe26f8704d8d"
format_stamp: "Formatted at 2026-09-24 07:12:08 on dist-test-slave-14ww"
server_key: "643c51f84e59fa453ea85241174da900"
server_key_iv: "712541acb897209945609f4eddf6e58b"
server_key_version: "encryptionkey@0"
I20260924 07:12:10.440563 15156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:10.455128 15156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:10.456933 15156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:10.457082 15156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:10.460558 15156 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35959
I20260924 07:12:10.460614 15223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35959 every 8 connection(s)
I20260924 07:12:10.461009 15156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:12:10.462311 15224 sys_catalog.cc:263] Verifying existing consensus state
I20260924 07:12:10.462869 15224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d: Bootstrap starting.
I20260924 07:12:10.469072 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 15156
I20260924 07:12:10.469417 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33485
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35959
--webserver_port=41621
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:12:10.471416 15224 log.cc:824] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d: Log is configured to *not* fsync() on all Append() calls
I20260924 07:12:10.473667 15224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:12:10.473954 15224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d: Bootstrap complete.
I20260924 07:12:10.476109 15224 raft_consensus.cc:348] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cffb68f1de7c4b2ea88bfe26f8704d8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35959 } }
I20260924 07:12:10.476313 15224 raft_consensus.cc:729] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cffb68f1de7c4b2ea88bfe26f8704d8d, State: Initialized, Role: FOLLOWER
I20260924 07:12:10.476470 15224 consensus_queue.cc:260] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cffb68f1de7c4b2ea88bfe26f8704d8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35959 } }
I20260924 07:12:10.476538 15224 raft_consensus.cc:388] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:12:10.476567 15224 raft_consensus.cc:482] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:12:10.476635 15224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [term 2 FOLLOWER]: Advancing to term 3
I20260924 07:12:10.477612 15224 raft_consensus.cc:504] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cffb68f1de7c4b2ea88bfe26f8704d8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35959 } }
I20260924 07:12:10.477696 15224 leader_election.cc:302] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cffb68f1de7c4b2ea88bfe26f8704d8d; no voters:
I20260924 07:12:10.477887 15224 leader_election.cc:288] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 07:12:10.477921 15228 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [term 3 FOLLOWER]: Leader election won for term 3
I20260924 07:12:10.478041 15228 raft_consensus.cc:686] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [term 3 LEADER]: Becoming Leader. State: Replica: cffb68f1de7c4b2ea88bfe26f8704d8d, State: Running, Role: LEADER
I20260924 07:12:10.478145 15224 sys_catalog.cc:564] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:12:10.478123 15228 consensus_queue.cc:237] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cffb68f1de7c4b2ea88bfe26f8704d8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35959 } }
I20260924 07:12:10.478406 15229 sys_catalog.cc:455] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "cffb68f1de7c4b2ea88bfe26f8704d8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cffb68f1de7c4b2ea88bfe26f8704d8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35959 } } }
I20260924 07:12:10.478503 15229 sys_catalog.cc:458] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [sys.catalog]: This master's current role is: LEADER
I20260924 07:12:10.478775 15230 sys_catalog.cc:455] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader cffb68f1de7c4b2ea88bfe26f8704d8d. Latest consensus state: current_term: 3 leader_uuid: "cffb68f1de7c4b2ea88bfe26f8704d8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cffb68f1de7c4b2ea88bfe26f8704d8d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35959 } } }
I20260924 07:12:10.478905 15230 sys_catalog.cc:458] T 00000000000000000000000000000000 P cffb68f1de7c4b2ea88bfe26f8704d8d [sys.catalog]: This master's current role is: LEADER
I20260924 07:12:10.479019 15233 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:12:10.479816 15233 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:12:10.479918 15233 catalog_manager.cc:1259] Loaded cluster ID: 140a175d6f164024ac21a3e82d632081
I20260924 07:12:10.479969 15233 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 07:12:10.480208 15233 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff0154239ad google::LogMessage::Fail() at ??:0
@ 0x7ff01542688a google::LogMessage::SendToLog() at ??:0
@ 0x7ff0154234de google::LogMessage::Flush() at ??:0
@ 0x7ff015424a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff01d3440f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7ff01d34d8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff0172f10d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff0172e6954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff01814a6db start_thread at ??:0
@ 0x7ff013f2161f clone at ??:0
W20260924 07:12:10.560593 15226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:10.560853 15226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:10.560889 15226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:10.562004 15226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:10.562070 15226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:10.562114 15226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:12:10.563566 15226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41621
--tserver_master_addrs=127.0.98.62:35959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:12:10.563822 15226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:10.564054 15226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:10.568969 15261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:10.569013 15226 server_base.cc:1034] running on GCE node
W20260924 07:12:10.568998 15263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:10.569096 15260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:10.569382 15226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:10.569650 15226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:10.570806 15226 hybrid_clock.cc:648] HybridClock initialized: now 1790233930570797 us; error 28 us; skew 500 ppm
Sep 24 07:12:10 dist-test-slave-14ww krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:12:10.573359 15226 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:12:10.573814 15226 webserver.cc:460] Webserver started at http://127.0.98.1:41621/ using document root <none> and password file <none>
I20260924 07:12:10.574064 15226 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:10.574136 15226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:10.575493 15226 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:12:10.576202 15270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:10.576387 15226 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:12:10.576478 15226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/wal
uuid: "26814bd24cdf410b9d59f11070bd7de3"
format_stamp: "Formatted at 2026-09-24 07:12:08 on dist-test-slave-14ww"
server_key: "917fea4663865ca7fdb4ab9031b87b75"
server_key_iv: "c3c358de4a1f2198501bb77d2de9c4a2"
server_key_version: "encryptionkey@0"
I20260924 07:12:10.576886 15226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:10.586055 15226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:10.588316 15226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:10.588475 15226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:10.588707 15226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:12:10.589229 15277 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 07:12:10.590627 15226 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 07:12:10.590675 15226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 07:12:10.590709 15226 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 07:12:10.591358 15226 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 07:12:10.591401 15226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 07:12:10.591451 15277 tablet_bootstrap.cc:492] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3: Bootstrap starting.
I20260924 07:12:10.599092 15226 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33485
I20260924 07:12:10.599248 15384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33485 every 8 connection(s)
I20260924 07:12:10.599615 15226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:12:10.602550 15277 log.cc:824] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3: Log is configured to *not* fsync() on all Append() calls
I20260924 07:12:10.604492 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 15226
I20260924 07:12:10.604684 15277 tablet_bootstrap.cc:492] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:12:10.604813 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42749
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35959
--webserver_port=34881
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:12:10.605062 15277 tablet_bootstrap.cc:492] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3: Bootstrap complete.
I20260924 07:12:10.606019 15277 ts_tablet_manager.cc:1397] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20260924 07:12:10.607178 15277 raft_consensus.cc:348] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c1d10160a245ccb85f17a8403be416" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 } } peers { permanent_uuid: "b0791f48efd344f580083d4d6eb70607" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 } } peers { permanent_uuid: "26814bd24cdf410b9d59f11070bd7de3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33485 } }
I20260924 07:12:10.607437 15277 raft_consensus.cc:729] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26814bd24cdf410b9d59f11070bd7de3, State: Initialized, Role: FOLLOWER
I20260924 07:12:10.607577 15277 consensus_queue.cc:260] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c1d10160a245ccb85f17a8403be416" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 } } peers { permanent_uuid: "b0791f48efd344f580083d4d6eb70607" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 } } peers { permanent_uuid: "26814bd24cdf410b9d59f11070bd7de3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33485 } }
I20260924 07:12:10.607878 15277 ts_tablet_manager.cc:1428] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260924 07:12:10.672700 15387 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:12:10.600286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.1:41203)
0924 07:12:10.600494 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:10.600550 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:10.601338 (+ 788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:10.672496 (+ 71158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":90,"threads_started":1}
W20260924 07:12:10.672987 15385 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35959 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35959: Client connection negotiation failed: client connection to 127.0.98.62:35959: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 07:12:10.673377 15387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:10.673249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.1:54707)
0924 07:12:10.673306 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:10.673353 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 07:12:10.673708 15387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:10.673613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.1:55897)
0924 07:12:10.673660 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:10.673688 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 07:12:10.673830 15385 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 07:12:10.687698 15389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:10.687880 15389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:10.687903 15389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:10.689127 15389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:10.689168 15389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:10.689218 15389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:12:10.690654 15389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42749
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34881
--tserver_master_addrs=127.0.98.62:35959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:12:10.690878 15389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:10.691107 15389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:10.696136 15396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:10.696197 15398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:10.696220 15395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:10.696229 15389 server_base.cc:1034] running on GCE node
I20260924 07:12:10.696530 15389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:10.696821 15389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:10.698011 15389 hybrid_clock.cc:648] HybridClock initialized: now 1790233930697995 us; error 24 us; skew 500 ppm
Sep 24 07:12:10 dist-test-slave-14ww krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:12:10.700567 15389 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:12:10.700981 15389 webserver.cc:460] Webserver started at http://127.0.98.2:34881/ using document root <none> and password file <none>
I20260924 07:12:10.701211 15389 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:10.701257 15389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:10.702615 15389 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:12:10.703316 15405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:10.703519 15389 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:12:10.703589 15389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/wal
uuid: "b0791f48efd344f580083d4d6eb70607"
format_stamp: "Formatted at 2026-09-24 07:12:09 on dist-test-slave-14ww"
server_key: "e86fd48a9bcd68d8f3bc8db61f62579d"
server_key_iv: "f2245b2c847cf066aed299f5d3b208c7"
server_key_version: "encryptionkey@0"
I20260924 07:12:10.703917 15389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:10.727500 15389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:10.729746 15389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:10.729909 15389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:10.730172 15389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:12:10.730491 15389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:12:10.730548 15389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:10.730592 15389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:12:10.730621 15389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:10.736570 15389 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42749
I20260924 07:12:10.736620 15518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42749 every 8 connection(s)
I20260924 07:12:10.736956 15389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:12:10.737978 15521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:10.737584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.2:33305)
0924 07:12:10.737735 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:10.737843 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":73,"threads_started":1}
W20260924 07:12:10.738387 15519 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35959 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35959: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
I20260924 07:12:10.738670 15521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:10.738577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.2:57201)
0924 07:12:10.738623 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:10.738650 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 07:12:10.738963 15521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:10.738889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.2:41901)
0924 07:12:10.738927 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:10.738950 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 07:12:10.739094 15519 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 07:12:10.741036 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 15389
I20260924 07:12:10.741292 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40703
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35959
--webserver_port=42137
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:12:10.823939 15522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:10.824157 15522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:10.824198 15522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:10.825448 15522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:10.825515 15522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:10.825557 15522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:12:10.826975 15522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40703
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42137
--tserver_master_addrs=127.0.98.62:35959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:12:10.827234 15522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:10.827451 15522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:10.832525 15528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:10.832563 15522 server_base.cc:1034] running on GCE node
W20260924 07:12:10.832530 15530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:10.832532 15527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:10.832991 15522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:10.833235 15522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:10.834384 15522 hybrid_clock.cc:648] HybridClock initialized: now 1790233930834371 us; error 28 us; skew 500 ppm
Sep 24 07:12:10 dist-test-slave-14ww krb5kdc[14085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:12:10.836719 15522 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:12:10.837184 15522 webserver.cc:460] Webserver started at http://127.0.98.3:42137/ using document root <none> and password file <none>
I20260924 07:12:10.837409 15522 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:10.837476 15522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:10.838826 15522 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:12:10.839427 15537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:10.839597 15522 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:12:10.839696 15522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/wal
uuid: "41c1d10160a245ccb85f17a8403be416"
format_stamp: "Formatted at 2026-09-24 07:12:09 on dist-test-slave-14ww"
server_key: "16a3564e3c8b7279ef844e12b0181f0d"
server_key_iv: "bb7d5fdee6d3917e055244b76b686df6"
server_key_version: "encryptionkey@0"
I20260924 07:12:10.840027 15522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:10.846715 15522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:10.848971 15522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:10.849120 15522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:10.849334 15522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:12:10.849656 15522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:12:10.849709 15522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:10.849750 15522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:12:10.849778 15522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:10.855276 15522 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40703
I20260924 07:12:10.855350 15650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40703 every 8 connection(s)
I20260924 07:12:10.855672 15522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:12:10.855862 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 15522
I20260924 07:12:10.856644 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:10.856277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.3:58533)
0924 07:12:10.856418 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:10.856487 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":81,"threads_started":1}
W20260924 07:12:10.857071 15651 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35959 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35959: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
I20260924 07:12:10.857336 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:10.857253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.3:47875)
0924 07:12:10.857295 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:10.857319 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 07:12:10.857621 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:10.857551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.3:55253)
0924 07:12:10.857588 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:10.857608 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 07:12:10.857746 15651 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 07:12:11.674469 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:11.674229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.1:60843)
0924 07:12:11.674364 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:11.674408 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":82,"threads_started":1}
I20260924 07:12:11.739684 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:11.739447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.2:52131)
0924 07:12:11.739590 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:11.739631 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":56,"threads_started":1}
I20260924 07:12:11.825836 15657 raft_consensus.cc:482] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:12:11.825996 15657 raft_consensus.cc:504] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c1d10160a245ccb85f17a8403be416" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 } } peers { permanent_uuid: "b0791f48efd344f580083d4d6eb70607" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 } } peers { permanent_uuid: "26814bd24cdf410b9d59f11070bd7de3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33485 } }
I20260924 07:12:11.826370 15657 leader_election.cc:288] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 41c1d10160a245ccb85f17a8403be416 (127.0.98.3:40703), b0791f48efd344f580083d4d6eb70607 (127.0.98.2:42749)
Sep 24 07:12:11 dist-test-slave-14ww krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 24 07:12:11 dist-test-slave-14ww krb5kdc[14085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260924 07:12:11.836930 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:11.826686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48221 (local address 127.0.98.2:42749)
0924 07:12:11.826899 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:11.826909 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:11.826946 (+ 37us) server_negotiation.cc:407] Connection header received
0924 07:12:11.827387 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:11.827389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:11.827427 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:11.827585 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:11.828327 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:11.829279 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:11.830234 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:11.830627 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:11.830749 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:11.833057 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:11.833060 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:11.833061 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:11.833071 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:11.835190 (+ 2119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:11.835714 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:11.835717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:11.835718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:11.835814 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:11.836070 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:11.836071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:11.836072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:11.836304 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:11.836398 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:11.836662 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 07:12:11.836813 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":60,"threads_started":1}
I20260924 07:12:11.836918 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:11.826561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42749 (local address 127.0.98.1:48221)
0924 07:12:11.826606 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:11.826617 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:11.826680 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:11.827617 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:11.827621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:11.828017 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:11.828195 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:11.828201 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:11.829400 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:11.829400 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:11.830135 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:11.830136 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:11.830728 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:11.830729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:11.830830 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:11.831412 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:11.831427 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:11.832956 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:12:11.835281 (+ 2325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:11.835287 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:11.835288 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:11.835497 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:11.835896 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:11.835897 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:11.835897 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:11.835997 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:11.836377 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:12:11.836383 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:12:11.836505 (+ 122us) client_negotiation.cc:769] Sending connection context
0924 07:12:11.836612 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 07:12:11.836823 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":3}
I20260924 07:12:11.837044 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:11.826613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40703 (local address 127.0.98.1:59425)
0924 07:12:11.826732 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:11.826742 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:11.826801 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:11.827618 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:11.827621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:11.828017 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:11.828195 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:11.828201 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:11.829360 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:11.829362 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:11.830138 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:11.830141 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:11.830715 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:11.830716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:11.830827 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:11.831480 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:11.831484 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:11.833364 (+ 1880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:12:11.835545 (+ 2181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:11.835546 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:11.835547 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:11.835724 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:11.835983 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:11.835984 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:11.835984 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:11.836048 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:11.836417 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:12:11.836418 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:12:11.836517 (+ 99us) client_negotiation.cc:769] Sending connection context
0924 07:12:11.836612 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 07:12:11.836904 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":210,"thread_start_us":49,"threads_started":1}
I20260924 07:12:11.837543 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:11.826849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59425 (local address 127.0.98.3:40703)
0924 07:12:11.826959 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:11.826967 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:11.826982 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:12:11.827380 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:11.827382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:11.827431 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:11.827581 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:11.828321 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:11.829262 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:11.830234 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:11.830623 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:11.830745 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:11.833485 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:11.833488 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:11.833489 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:11.833498 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:11.835462 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:11.835814 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:11.835816 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:11.835817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:11.835906 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:11.836114 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:11.836115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:11.836116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:11.836343 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:11.836434 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:11.836666 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 07:12:11.836857 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":39,"threads_started":1}
W20260924 07:12:11.837723 15271 leader_election.cc:341] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b0791f48efd344f580083d4d6eb70607 (127.0.98.2:42749): Not found: Tablet not found: c4dc70c6285042078d1adbae705bf067
W20260924 07:12:11.838413 15273 leader_election.cc:341] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 41c1d10160a245ccb85f17a8403be416 (127.0.98.3:40703): Not found: Tablet not found: c4dc70c6285042078d1adbae705bf067
I20260924 07:12:11.838495 15273 leader_election.cc:302] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26814bd24cdf410b9d59f11070bd7de3; no voters: 41c1d10160a245ccb85f17a8403be416, b0791f48efd344f580083d4d6eb70607
I20260924 07:12:11.838624 15657 raft_consensus.cc:2726] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:12:11.858309 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:11.858099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.3:60591)
0924 07:12:11.858229 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:11.858277 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":64,"threads_started":1}
I20260924 07:12:12.675153 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:12.674959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.1:46497)
0924 07:12:12.675078 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:12.675109 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260924 07:12:12.740319 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:12.740138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.2:43063)
0924 07:12:12.740258 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:12.740289 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":71,"threads_started":1}
I20260924 07:12:12.858894 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:12.858690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.3:47199)
0924 07:12:12.858811 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:12.858843 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
I20260924 07:12:13.482933 15665 raft_consensus.cc:482] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:12:13.483026 15665 raft_consensus.cc:504] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c1d10160a245ccb85f17a8403be416" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 } } peers { permanent_uuid: "b0791f48efd344f580083d4d6eb70607" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 } } peers { permanent_uuid: "26814bd24cdf410b9d59f11070bd7de3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33485 } }
I20260924 07:12:13.483248 15665 leader_election.cc:288] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 41c1d10160a245ccb85f17a8403be416 (127.0.98.3:40703), b0791f48efd344f580083d4d6eb70607 (127.0.98.2:42749)
W20260924 07:12:13.483654 15271 leader_election.cc:341] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b0791f48efd344f580083d4d6eb70607 (127.0.98.2:42749): Not found: Tablet not found: c4dc70c6285042078d1adbae705bf067
W20260924 07:12:13.483731 15273 leader_election.cc:341] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 41c1d10160a245ccb85f17a8403be416 (127.0.98.3:40703): Not found: Tablet not found: c4dc70c6285042078d1adbae705bf067
I20260924 07:12:13.483806 15273 leader_election.cc:302] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26814bd24cdf410b9d59f11070bd7de3; no voters: 41c1d10160a245ccb85f17a8403be416, b0791f48efd344f580083d4d6eb70607
I20260924 07:12:13.483892 15665 raft_consensus.cc:2726] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:12:13.675786 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:13.675594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.1:53735)
0924 07:12:13.675702 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:13.675738 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":63,"threads_started":1}
I20260924 07:12:13.740976 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:13.740785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.2:35925)
0924 07:12:13.740905 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:13.740939 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
I20260924 07:12:13.859532 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:13.859345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.3:55477)
0924 07:12:13.859458 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:13.859491 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
I20260924 07:12:14.676450 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:14.676202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.1:50019)
0924 07:12:14.676367 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:14.676403 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":78,"threads_started":1}
I20260924 07:12:14.741621 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:14.741399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.2:33801)
0924 07:12:14.741536 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:14.741572 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":59,"threads_started":1}
I20260924 07:12:14.860174 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:14.859991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.3:59759)
0924 07:12:14.860106 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:14.860139 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":59,"threads_started":1}
I20260924 07:12:15.324888 15672 raft_consensus.cc:482] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:12:15.324982 15672 raft_consensus.cc:504] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c1d10160a245ccb85f17a8403be416" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 } } peers { permanent_uuid: "b0791f48efd344f580083d4d6eb70607" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 } } peers { permanent_uuid: "26814bd24cdf410b9d59f11070bd7de3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33485 } }
I20260924 07:12:15.325191 15672 leader_election.cc:288] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 41c1d10160a245ccb85f17a8403be416 (127.0.98.3:40703), b0791f48efd344f580083d4d6eb70607 (127.0.98.2:42749)
W20260924 07:12:15.325538 15271 leader_election.cc:341] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b0791f48efd344f580083d4d6eb70607 (127.0.98.2:42749): Not found: Tablet not found: c4dc70c6285042078d1adbae705bf067
W20260924 07:12:15.325603 15273 leader_election.cc:341] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 41c1d10160a245ccb85f17a8403be416 (127.0.98.3:40703): Not found: Tablet not found: c4dc70c6285042078d1adbae705bf067
I20260924 07:12:15.325661 15273 leader_election.cc:302] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26814bd24cdf410b9d59f11070bd7de3; no voters: 41c1d10160a245ccb85f17a8403be416, b0791f48efd344f580083d4d6eb70607
I20260924 07:12:15.325736 15672 raft_consensus.cc:2726] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:12:15.677165 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:15.676955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.1:35535)
0924 07:12:15.677083 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:15.677117 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":61,"threads_started":1}
I20260924 07:12:15.742251 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:15.742051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.2:54489)
0924 07:12:15.742181 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:15.742215 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":69,"threads_started":1}
I20260924 07:12:15.861027 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:15.860725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.3:44893)
0924 07:12:15.860946 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:15.860984 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":84,"threads_started":1}
I20260924 07:12:16.677942 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:16.677729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.1:33233)
0924 07:12:16.677865 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:16.677897 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
I20260924 07:12:16.742993 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:16.742735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.2:57651)
0924 07:12:16.742888 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:16.742924 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":62,"threads_started":1}
I20260924 07:12:16.861867 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:16.861627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.3:51775)
0924 07:12:16.861786 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:16.861824 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":81,"threads_started":1}
I20260924 07:12:17.678577 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:17.678360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.1:52217)
0924 07:12:17.678496 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:17.678531 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":61,"threads_started":1}
I20260924 07:12:17.743587 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:17.743403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.2:40501)
0924 07:12:17.743515 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:17.743548 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":57,"threads_started":1}
I20260924 07:12:17.862545 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:17.862318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.3:36909)
0924 07:12:17.862462 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:17.862499 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":72,"threads_started":1}
I20260924 07:12:18.679370 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:18.679111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.1:55885)
0924 07:12:18.679285 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:18.679322 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":86,"threads_started":1}
I20260924 07:12:18.744256 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:18.744015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.2:44361)
0924 07:12:18.744179 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:18.744214 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":85,"threads_started":1}
I20260924 07:12:18.863210 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:18.863001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.3:53705)
0924 07:12:18.863135 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:18.863169 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":72,"threads_started":1}
I20260924 07:12:19.680020 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:19.679825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.1:48305)
0924 07:12:19.679937 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:19.679971 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":57,"threads_started":1}
I20260924 07:12:19.744892 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:19.744644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.2:41687)
0924 07:12:19.744823 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:19.744856 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":60,"threads_started":1}
I20260924 07:12:19.863888 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:19.863679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.3:60545)
0924 07:12:19.863811 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:19.863847 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
I20260924 07:12:20.142928 15688 raft_consensus.cc:482] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:12:20.143020 15688 raft_consensus.cc:504] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c1d10160a245ccb85f17a8403be416" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 } } peers { permanent_uuid: "b0791f48efd344f580083d4d6eb70607" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 } } peers { permanent_uuid: "26814bd24cdf410b9d59f11070bd7de3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33485 } }
I20260924 07:12:20.143177 15688 leader_election.cc:288] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 41c1d10160a245ccb85f17a8403be416 (127.0.98.3:40703), b0791f48efd344f580083d4d6eb70607 (127.0.98.2:42749)
W20260924 07:12:20.143523 15273 leader_election.cc:341] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 41c1d10160a245ccb85f17a8403be416 (127.0.98.3:40703): Not found: Tablet not found: c4dc70c6285042078d1adbae705bf067
W20260924 07:12:20.143595 15271 leader_election.cc:341] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b0791f48efd344f580083d4d6eb70607 (127.0.98.2:42749): Not found: Tablet not found: c4dc70c6285042078d1adbae705bf067
I20260924 07:12:20.143625 15273 leader_election.cc:302] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26814bd24cdf410b9d59f11070bd7de3; no voters: 41c1d10160a245ccb85f17a8403be416, b0791f48efd344f580083d4d6eb70607
I20260924 07:12:20.143711 15688 raft_consensus.cc:2726] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:12:20.680830 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:20.680578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.1:39901)
0924 07:12:20.680699 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:20.680786 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260924 07:12:20.745544 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:20.745347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.2:37097)
0924 07:12:20.745469 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:20.745505 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":58,"threads_started":1}
I20260924 07:12:20.864507 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:20.864304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.3:43071)
0924 07:12:20.864435 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:20.864467 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
I20260924 07:12:21.659652 15692 raft_consensus.cc:482] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:12:21.659780 15692 raft_consensus.cc:504] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41c1d10160a245ccb85f17a8403be416" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 } } peers { permanent_uuid: "b0791f48efd344f580083d4d6eb70607" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42749 } } peers { permanent_uuid: "26814bd24cdf410b9d59f11070bd7de3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33485 } }
I20260924 07:12:21.659997 15692 leader_election.cc:288] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 41c1d10160a245ccb85f17a8403be416 (127.0.98.3:40703), b0791f48efd344f580083d4d6eb70607 (127.0.98.2:42749)
W20260924 07:12:21.660491 15273 leader_election.cc:341] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 41c1d10160a245ccb85f17a8403be416 (127.0.98.3:40703): Not found: Tablet not found: c4dc70c6285042078d1adbae705bf067
W20260924 07:12:21.660561 15271 leader_election.cc:341] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b0791f48efd344f580083d4d6eb70607 (127.0.98.2:42749): Not found: Tablet not found: c4dc70c6285042078d1adbae705bf067
I20260924 07:12:21.660621 15271 leader_election.cc:302] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26814bd24cdf410b9d59f11070bd7de3; no voters: 41c1d10160a245ccb85f17a8403be416, b0791f48efd344f580083d4d6eb70607
I20260924 07:12:21.660715 15692 raft_consensus.cc:2726] T c4dc70c6285042078d1adbae705bf067 P 26814bd24cdf410b9d59f11070bd7de3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:12:21.681442 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:21.681257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.1:52571)
0924 07:12:21.681376 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:21.681407 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":69,"threads_started":1}
I20260924 07:12:21.746173 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:21.745979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.2:45201)
0924 07:12:21.746105 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:21.746137 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":70,"threads_started":1}
I20260924 07:12:21.865173 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:21.864911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.3:37515)
0924 07:12:21.865062 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:21.865096 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":73,"threads_started":1}
I20260924 07:12:22.682143 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:22.681897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.1:39343)
0924 07:12:22.682027 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:22.682062 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":55,"threads_started":1}
I20260924 07:12:22.746784 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:22.746585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.2:43181)
0924 07:12:22.746712 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:22.746749 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":56,"threads_started":1}
I20260924 07:12:22.865799 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:22.865605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.3:44655)
0924 07:12:22.865728 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:22.865761 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":57,"threads_started":1}
I20260924 07:12:23.682936 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:23.682714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.1:40611)
0924 07:12:23.682856 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:23.682889 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
I20260924 07:12:23.747392 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:23.747218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.2:53707)
0924 07:12:23.747322 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:23.747355 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":53,"threads_started":1}
I20260924 07:12:23.866425 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:23.866219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.3:41773)
0924 07:12:23.866352 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:23.866386 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":60,"threads_started":1}
I20260924 07:12:24.683569 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:24.683349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.1:45365)
0924 07:12:24.683479 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:24.683512 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":58,"threads_started":1}
I20260924 07:12:24.748000 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:24.747778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.2:59599)
0924 07:12:24.747926 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:24.747960 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":79,"threads_started":1}
I20260924 07:12:24.867147 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:24.866865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.3:52527)
0924 07:12:24.867068 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:24.867105 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":79,"threads_started":1}
I20260924 07:12:25.684340 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:25.684128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.1:49801)
0924 07:12:25.684265 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:25.684294 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
I20260924 07:12:25.748900 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:25.748544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.2:37587)
0924 07:12:25.748788 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:25.748837 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":78,"threads_started":1}
I20260924 07:12:25.861552 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 15226
I20260924 07:12:25.866516 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 15389
I20260924 07:12:25.868379 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:25.868191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35959 (local address 127.0.98.3:47955)
0924 07:12:25.868313 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:25.868341 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":72,"threads_started":1}
I20260924 07:12:25.870955 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 15522
2026-09-24T07:12:25Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18446 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:12:25 dist-test-slave-14ww krb5kdc[15710](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:12:25 dist-test-slave-14ww krb5kdc[15710](info): set up 2 sockets
Sep 24 07:12:25 dist-test-slave-14ww krb5kdc[15710](info): commencing operation
krb5kdc: starting...
W20260924 07:12:26.908835 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:12:26 dist-test-slave-14ww krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:12:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:12:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:12:27.025839 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37127
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:12:27.103576 15723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:27.103760 15723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:27.103782 15723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:27.104894 15723 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 07:12:27.104930 15723 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 07:12:27.104944 15723 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 07:12:27.104955 15723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:12:27.104969 15723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:27.104981 15723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:27.104993 15723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:12:27.105006 15723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:12:27.106370 15723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:12:27.106540 15723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:27.106691 15723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:27.111614 15728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:27.111711 15723 server_base.cc:1034] running on GCE node
W20260924 07:12:27.111651 15731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:27.111723 15729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:27.112052 15723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:27.112317 15723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:27.113507 15723 hybrid_clock.cc:648] HybridClock initialized: now 1790233947113485 us; error 36 us; skew 500 ppm
Sep 24 07:12:27 dist-test-slave-14ww krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:12:27.115759 15723 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:12:27.116197 15723 webserver.cc:460] Webserver started at http://127.0.98.62:41239/ using document root <none> and password file <none>
I20260924 07:12:27.116412 15723 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:27.116478 15723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:27.116623 15723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:12:27.117673 15723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "fc6625c0c7d047e594a80ff678e26bdc"
format_stamp: "Formatted at 2026-09-24 07:12:27 on dist-test-slave-14ww"
server_key: "782ee73060c760169f463f8057bac125"
server_key_iv: "086b8c1a314427f9c102325a87a593d4"
server_key_version: "encryptionkey@0"
I20260924 07:12:27.118022 15723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "fc6625c0c7d047e594a80ff678e26bdc"
format_stamp: "Formatted at 2026-09-24 07:12:27 on dist-test-slave-14ww"
server_key: "782ee73060c760169f463f8057bac125"
server_key_iv: "086b8c1a314427f9c102325a87a593d4"
server_key_version: "encryptionkey@0"
I20260924 07:12:27.119271 15723 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:12:27.119997 15738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:27.120194 15723 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:12:27.120301 15723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "fc6625c0c7d047e594a80ff678e26bdc"
format_stamp: "Formatted at 2026-09-24 07:12:27 on dist-test-slave-14ww"
server_key: "782ee73060c760169f463f8057bac125"
server_key_iv: "086b8c1a314427f9c102325a87a593d4"
server_key_version: "encryptionkey@0"
I20260924 07:12:27.120389 15723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:27.135479 15723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:27.137229 15723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:27.137379 15723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:27.141217 15723 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37127
I20260924 07:12:27.141251 15790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37127 every 8 connection(s)
I20260924 07:12:27.141583 15723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:12:27.142356 15791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:12:27.144382 15791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc: Bootstrap starting.
I20260924 07:12:27.144961 15791 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc: Neither blocks nor log segments found. Creating new log.
I20260924 07:12:27.145193 15791 log.cc:824] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc: Log is configured to *not* fsync() on all Append() calls
I20260924 07:12:27.145826 15791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc: No bootstrap required, opened a new log
I20260924 07:12:27.147256 15791 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6625c0c7d047e594a80ff678e26bdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37127 } }
I20260924 07:12:27.147382 15791 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:12:27.147403 15791 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc6625c0c7d047e594a80ff678e26bdc, State: Initialized, Role: FOLLOWER
I20260924 07:12:27.147531 15791 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6625c0c7d047e594a80ff678e26bdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37127 } }
I20260924 07:12:27.147607 15791 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:12:27.147630 15791 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:12:27.147657 15791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:12:27.148114 15791 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6625c0c7d047e594a80ff678e26bdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37127 } }
I20260924 07:12:27.148195 15791 leader_election.cc:302] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc6625c0c7d047e594a80ff678e26bdc; no voters:
I20260924 07:12:27.148309 15791 leader_election.cc:288] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:12:27.148375 15794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:12:27.148572 15791 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:12:27.148600 15794 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [term 1 LEADER]: Becoming Leader. State: Replica: fc6625c0c7d047e594a80ff678e26bdc, State: Running, Role: LEADER
I20260924 07:12:27.148694 15794 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6625c0c7d047e594a80ff678e26bdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37127 } }
I20260924 07:12:27.149139 15796 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc6625c0c7d047e594a80ff678e26bdc. Latest consensus state: current_term: 1 leader_uuid: "fc6625c0c7d047e594a80ff678e26bdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6625c0c7d047e594a80ff678e26bdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37127 } } }
I20260924 07:12:27.149179 15795 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc6625c0c7d047e594a80ff678e26bdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6625c0c7d047e594a80ff678e26bdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37127 } } }
I20260924 07:12:27.149335 15796 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [sys.catalog]: This master's current role is: LEADER
I20260924 07:12:27.149348 15795 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [sys.catalog]: This master's current role is: LEADER
I20260924 07:12:27.149708 15803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:12:27.150058 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 15723
I20260924 07:12:27.150157 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:12:27.150270 15803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:12:27.150362 392 external_mini_cluster.cc:1428] Setting key 5204cd1a4aed4a3cb56c15aa7d90eb0f
I20260924 07:12:27.152333 15803 catalog_manager.cc:1347] Generated new cluster ID: 206e4d8a723e4869b8c0297eab5c66d0
I20260924 07:12:27.152390 15803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 07:12:27 dist-test-slave-14ww krb5kdc[15710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:12:27.161899 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:27.150784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49780 (local address 127.0.98.62:37127)
0924 07:12:27.151104 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:27.151112 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:27.151142 (+ 30us) server_negotiation.cc:407] Connection header received
0924 07:12:27.151239 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:27.151244 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:27.151283 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:27.151420 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:27.152304 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:27.153702 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:27.154780 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:27.155352 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:27.155444 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:27.158231 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:27.158236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:27.158236 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:27.158248 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:27.160222 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:27.160628 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:27.160631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:27.160631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:27.160761 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:27.160992 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:27.160992 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:27.160993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:27.161293 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:27.161432 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:27.161527 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 07:12:27.161662 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":71,"threads_started":1}
I20260924 07:12:27.165727 15803 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:12:27.166183 15803 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:12:27.170250 15803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc: Generated new TSK 0
I20260924 07:12:27.170418 15803 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 07:12:27.199724 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37127
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:12:27.279552 15817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:27.279747 15817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:27.279768 15817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:27.280923 15817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:27.280962 15817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:27.281010 15817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:12:27.282398 15817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:12:27.282626 15817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:27.282850 15817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:27.287797 15822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:27.287948 15823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:27.287825 15825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:27.288059 15817 server_base.cc:1034] running on GCE node
I20260924 07:12:27.288259 15817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:27.288493 15817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:27.289654 15817 hybrid_clock.cc:648] HybridClock initialized: now 1790233947289643 us; error 29 us; skew 500 ppm
Sep 24 07:12:27 dist-test-slave-14ww krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:12:27.292155 15817 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:12:27.292586 15817 webserver.cc:460] Webserver started at http://127.0.98.1:36427/ using document root <none> and password file <none>
I20260924 07:12:27.292814 15817 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:27.292862 15817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:27.292948 15817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:12:27.293855 15817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/data/instance:
uuid: "54431bf7bec5412483921836103ad70d"
format_stamp: "Formatted at 2026-09-24 07:12:27 on dist-test-slave-14ww"
server_key: "49aeb66134fa49a1842c4c882ef8d5a7"
server_key_iv: "f4d1f6c520ee3b1b71fc447996c32afd"
server_key_version: "encryptionkey@0"
I20260924 07:12:27.294137 15817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/wal/instance:
uuid: "54431bf7bec5412483921836103ad70d"
format_stamp: "Formatted at 2026-09-24 07:12:27 on dist-test-slave-14ww"
server_key: "49aeb66134fa49a1842c4c882ef8d5a7"
server_key_iv: "f4d1f6c520ee3b1b71fc447996c32afd"
server_key_version: "encryptionkey@0"
I20260924 07:12:27.295311 15817 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 07:12:27.295962 15832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:27.296114 15817 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:12:27.296193 15817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/wal
uuid: "54431bf7bec5412483921836103ad70d"
format_stamp: "Formatted at 2026-09-24 07:12:27 on dist-test-slave-14ww"
server_key: "49aeb66134fa49a1842c4c882ef8d5a7"
server_key_iv: "f4d1f6c520ee3b1b71fc447996c32afd"
server_key_version: "encryptionkey@0"
I20260924 07:12:27.296245 15817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:27.318976 15817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:27.320873 15817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:27.321022 15817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:27.321250 15817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:12:27.321555 15817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:12:27.321587 15817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:27.321641 15817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:12:27.321666 15817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:27.327363 15817 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43979
I20260924 07:12:27.327436 15945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43979 every 8 connection(s)
I20260924 07:12:27.327744 15817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 07:12:27 dist-test-slave-14ww krb5kdc[15710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:12:27.334272 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 15817
I20260924 07:12:27.334384 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:12:27.334582 392 external_mini_cluster.cc:1428] Setting key 63849c4b1ed0638bae0666a204d2ff8d
I20260924 07:12:27.336766 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:27.328478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48771 (local address 127.0.98.62:37127)
0924 07:12:27.328523 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:27.328527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:27.328773 (+ 246us) server_negotiation.cc:407] Connection header received
0924 07:12:27.329236 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:27.329237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:27.329250 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:27.329304 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:27.330180 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:27.330677 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:27.331600 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:27.331810 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:27.331866 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:27.333818 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:27.333819 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:27.333819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:27.333825 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:27.335403 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:27.335798 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:27.335799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:27.335800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:27.335867 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:27.336110 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:27.336111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:27.336111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:27.336229 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:27.336305 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:27.336569 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 07:12:27.336650 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 07:12:27.336928 15948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:27.328467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.1:48771)
0924 07:12:27.328624 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:27.328656 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:27.329153 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:27.329356 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:27.329359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:27.329898 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:27.330075 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:27.330080 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:27.330757 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:27.330757 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:27.331525 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:27.331527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:27.331886 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:27.331886 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:27.331979 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:27.332396 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:27.332404 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:27.333731 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:12:27.335491 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:27.335498 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:27.335500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:27.335714 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:27.335934 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:27.335935 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:27.335935 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:27.336041 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:27.336297 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:12:27.336303 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:12:27.336441 (+ 138us) client_negotiation.cc:769] Sending connection context
0924 07:12:27.336546 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 07:12:27.336690 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":86,"threads_started":1}
I20260924 07:12:27.337396 15946 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37127
I20260924 07:12:27.337469 15946 heartbeater.cc:464] Registering TS with master...
I20260924 07:12:27.337654 15946 heartbeater.cc:511] Master 127.0.98.62:37127 requested a full tablet report, sending...
I20260924 07:12:27.338119 15755 ts_manager.cc:194] Registered new tserver with Master: 54431bf7bec5412483921836103ad70d (127.0.98.1:43979)
I20260924 07:12:27.338697 15755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48771
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 07:12:27.359077 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37127
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:12:27.437559 15951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:27.437749 15951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:27.437772 15951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:27.438899 15951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:27.438938 15951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:27.438963 15951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:12:27.440378 15951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:12:27.440594 15951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:27.440868 15951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:27.445719 15956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:27.445816 15959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:27.445740 15951 server_base.cc:1034] running on GCE node
W20260924 07:12:27.445747 15957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:27.446091 15951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:27.446372 15951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:27.447561 15951 hybrid_clock.cc:648] HybridClock initialized: now 1790233947447559 us; error 40 us; skew 500 ppm
Sep 24 07:12:27 dist-test-slave-14ww krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:12:27.450430 15951 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:12:27.450846 15951 webserver.cc:460] Webserver started at http://127.0.98.2:45559/ using document root <none> and password file <none>
I20260924 07:12:27.451072 15951 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:27.451117 15951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:27.451259 15951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:12:27.452271 15951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/data/instance:
uuid: "67d8d14af05940f0b8408e3b8162d1a1"
format_stamp: "Formatted at 2026-09-24 07:12:27 on dist-test-slave-14ww"
server_key: "d33af9cf35d87548fe4aa025c70e1d79"
server_key_iv: "5c5a48d2cdde09b55f42b9ef12d62fb3"
server_key_version: "encryptionkey@0"
I20260924 07:12:27.452629 15951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/wal/instance:
uuid: "67d8d14af05940f0b8408e3b8162d1a1"
format_stamp: "Formatted at 2026-09-24 07:12:27 on dist-test-slave-14ww"
server_key: "d33af9cf35d87548fe4aa025c70e1d79"
server_key_iv: "5c5a48d2cdde09b55f42b9ef12d62fb3"
server_key_version: "encryptionkey@0"
I20260924 07:12:27.453961 15951 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:12:27.454692 15966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:27.454905 15951 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:12:27.455005 15951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/wal
uuid: "67d8d14af05940f0b8408e3b8162d1a1"
format_stamp: "Formatted at 2026-09-24 07:12:27 on dist-test-slave-14ww"
server_key: "d33af9cf35d87548fe4aa025c70e1d79"
server_key_iv: "5c5a48d2cdde09b55f42b9ef12d62fb3"
server_key_version: "encryptionkey@0"
I20260924 07:12:27.455077 15951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:27.469038 15951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:27.471336 15951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:27.471498 15951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:27.471740 15951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:12:27.472045 15951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:12:27.472079 15951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:27.472101 15951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:12:27.472148 15951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:27.477864 15951 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35475
I20260924 07:12:27.477908 16079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35475 every 8 connection(s)
I20260924 07:12:27.478209 15951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:12:27.483302 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 15951
I20260924 07:12:27.483423 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:12:27.483639 392 external_mini_cluster.cc:1428] Setting key f910d3e51ff25f62d4608a0fed243753
Sep 24 07:12:27 dist-test-slave-14ww krb5kdc[15710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:12:27.487846 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:27.478875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37955 (local address 127.0.98.62:37127)
0924 07:12:27.478921 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:27.478923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:27.479109 (+ 186us) server_negotiation.cc:407] Connection header received
0924 07:12:27.479585 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:27.479586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:27.479596 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:27.479659 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:27.480564 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:27.481096 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:27.482184 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:27.482402 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:27.482462 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:27.484771 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:27.484772 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:27.484772 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:27.484779 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:27.486486 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:27.486869 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:27.486871 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:27.486871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:27.486936 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:27.487165 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:27.487166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:27.487166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:27.487315 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:27.487385 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:27.487612 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 07:12:27.487718 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 07:12:27.487859 16082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:27.478817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.2:37955)
0924 07:12:27.478972 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:27.479012 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:27.479500 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:27.479737 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:27.479740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:27.480286 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:27.480464 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:27.480468 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:27.481180 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:27.481181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:27.482077 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:27.482079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:27.482482 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:27.482483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:27.482578 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:27.482985 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:27.482993 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:27.484644 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:12:27.486572 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:27.486577 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:27.486578 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:27.486794 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:27.487004 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:27.487005 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:27.487005 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:27.487094 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:27.487380 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:12:27.487383 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:12:27.487485 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 07:12:27.487562 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 07:12:27.487673 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260924 07:12:27.488497 16080 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37127
I20260924 07:12:27.488597 16080 heartbeater.cc:464] Registering TS with master...
I20260924 07:12:27.488822 16080 heartbeater.cc:511] Master 127.0.98.62:37127 requested a full tablet report, sending...
I20260924 07:12:27.489259 15755 ts_manager.cc:194] Registered new tserver with Master: 67d8d14af05940f0b8408e3b8162d1a1 (127.0.98.2:35475)
I20260924 07:12:27.489624 15755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37955
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 07:12:27.509590 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37127
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:12:27.587561 16085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:27.587745 16085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:27.587767 16085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:27.588897 16085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:27.588935 16085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:27.588960 16085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:12:27.590458 16085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:12:27.590684 16085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:27.590941 16085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:27.595732 16090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:27.595758 16093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:27.595793 16085 server_base.cc:1034] running on GCE node
W20260924 07:12:27.595758 16091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:27.596185 16085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:27.596484 16085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:27.597641 16085 hybrid_clock.cc:648] HybridClock initialized: now 1790233947597632 us; error 27 us; skew 500 ppm
Sep 24 07:12:27 dist-test-slave-14ww krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:12:27.600215 16085 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:12:27.600562 16085 webserver.cc:460] Webserver started at http://127.0.98.3:33445/ using document root <none> and password file <none>
I20260924 07:12:27.600772 16085 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:27.600834 16085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:27.600922 16085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:12:27.601823 16085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/data/instance:
uuid: "5ad5251214014400b3fb8f34d80c0363"
format_stamp: "Formatted at 2026-09-24 07:12:27 on dist-test-slave-14ww"
server_key: "6d835bf92972584cf86fe74a0ff83cad"
server_key_iv: "9cbcd799cca54ea60b5af819b191b64a"
server_key_version: "encryptionkey@0"
I20260924 07:12:27.602108 16085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5ad5251214014400b3fb8f34d80c0363"
format_stamp: "Formatted at 2026-09-24 07:12:27 on dist-test-slave-14ww"
server_key: "6d835bf92972584cf86fe74a0ff83cad"
server_key_iv: "9cbcd799cca54ea60b5af819b191b64a"
server_key_version: "encryptionkey@0"
I20260924 07:12:27.603327 16085 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:12:27.604005 16100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:27.604174 16085 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:12:27.604239 16085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/wal
uuid: "5ad5251214014400b3fb8f34d80c0363"
format_stamp: "Formatted at 2026-09-24 07:12:27 on dist-test-slave-14ww"
server_key: "6d835bf92972584cf86fe74a0ff83cad"
server_key_iv: "9cbcd799cca54ea60b5af819b191b64a"
server_key_version: "encryptionkey@0"
I20260924 07:12:27.604288 16085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:27.613281 16085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:27.615339 16085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:27.615442 16085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:27.615586 16085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:12:27.615804 16085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:12:27.615832 16085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:27.615855 16085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:12:27.615868 16085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:27.621280 16085 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45897
I20260924 07:12:27.621346 16213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45897 every 8 connection(s)
I20260924 07:12:27.621626 16085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:12:27.623428 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 16085
I20260924 07:12:27.623533 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:12:27.623737 392 external_mini_cluster.cc:1428] Setting key 47a971d303587266d245cd6025d21687
Sep 24 07:12:27 dist-test-slave-14ww krb5kdc[15710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:12:27.630357 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:27.622292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34283 (local address 127.0.98.62:37127)
0924 07:12:27.622364 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:27.622366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:27.622519 (+ 153us) server_negotiation.cc:407] Connection header received
0924 07:12:27.622934 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:27.622935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:27.622945 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:27.622995 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:27.624199 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:27.624969 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:27.625884 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:27.626058 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:27.626105 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:27.627940 (+ 1835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:27.627941 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:27.627942 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:27.627947 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:27.629125 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:27.629532 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:27.629533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:27.629534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:27.629592 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:27.629812 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:27.629813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:27.629813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:27.629927 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:27.629983 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:27.630230 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 07:12:27.630294 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 07:12:27.630468 16216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:27.622270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.3:34283)
0924 07:12:27.622404 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:27.622435 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:27.622856 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:27.623041 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:27.623044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:27.623808 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:27.624075 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:27.624096 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:27.625042 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:27.625043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:27.625814 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:27.625816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:27.626123 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:27.626124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:27.626216 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:27.626593 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:27.626600 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:27.627862 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:12:27.629208 (+ 1346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:27.629213 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:27.629214 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:27.629419 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:27.629660 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:27.629661 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:27.629661 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:27.629748 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:27.629986 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:12:27.629990 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:12:27.630089 (+ 99us) client_negotiation.cc:769] Sending connection context
0924 07:12:27.630172 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 07:12:27.630278 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":79,"threads_started":1}
I20260924 07:12:27.630977 16214 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37127
I20260924 07:12:27.631068 16214 heartbeater.cc:464] Registering TS with master...
I20260924 07:12:27.631196 16214 heartbeater.cc:511] Master 127.0.98.62:37127 requested a full tablet report, sending...
I20260924 07:12:27.631542 15755 ts_manager.cc:194] Registered new tserver with Master: 5ad5251214014400b3fb8f34d80c0363 (127.0.98.3:45897)
I20260924 07:12:27.631903 15755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34283
I20260924 07:12:27.635288 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:12:27.643502 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:27.636940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49784 (local address 127.0.98.62:37127)
0924 07:12:27.636998 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:27.637000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:27.637140 (+ 140us) server_negotiation.cc:407] Connection header received
0924 07:12:27.637209 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:27.637209 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:27.637239 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:27.637295 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:27.637792 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:27.638269 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:27.639010 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:27.639187 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:27.639251 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:27.640539 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:27.640541 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:27.640541 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:27.640547 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:27.642270 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:27.642700 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:27.642701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:27.642701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:27.642756 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:27.642973 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:27.642974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:27.642974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:27.643114 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:27.643215 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:27.643329 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 07:12:27.643403 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 07:12:27.646503 15755 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:12:27.648627 15755 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:12:27.654903 16225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:27.650809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35475 (local address 127.0.0.1:45876)
0924 07:12:27.651179 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:27.651188 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:27.651239 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:27.651458 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:27.651461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:27.651463 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:12:27.651529 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:27.651530 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:27.652946 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:27.652947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:27.653690 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:27.653692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:27.654570 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:27.654571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:27.654661 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:27.654673 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 07:12:27.654779 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 07:12:27.654834 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":49,"threads_started":1}
I20260924 07:12:27.654912 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:27.651017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45876 (local address 127.0.98.2:35475)
0924 07:12:27.651161 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:27.651166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:27.651234 (+ 68us) server_negotiation.cc:407] Connection header received
0924 07:12:27.651317 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:27.651319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:27.651337 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:27.651422 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:12:27.651687 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:27.652831 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:27.653791 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:27.654476 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:27.654567 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:27.654624 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:27.654797 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 07:12:27.654834 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
I20260924 07:12:27.654994 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:27.650976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43979 (local address 127.0.0.1:36794)
0924 07:12:27.651193 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:27.651204 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:27.651259 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:27.651522 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:27.651523 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:27.651524 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:12:27.651594 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:27.651596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:27.652578 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:27.652579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:27.653894 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:27.653897 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:27.654762 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:27.654764 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:27.654824 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:27.654829 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:12:27.654881 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 07:12:27.654923 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":43,"thread_start_us":29,"threads_started":1}
I20260924 07:12:27.655318 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:27.651163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36794 (local address 127.0.98.1:43979)
0924 07:12:27.651366 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:27.651370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:27.651377 (+ 7us) server_negotiation.cc:407] Connection header received
0924 07:12:27.651431 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:27.651433 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:27.651449 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:27.651528 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:12:27.651714 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:27.652475 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:27.654071 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:27.654660 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:27.654758 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:27.654812 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:27.655057 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 07:12:27.655237 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":78,"threads_started":1}
I20260924 07:12:27.655764 16014 tablet_service.cc:1467] Processing CreateTablet for tablet fd61772f63324aa38d5a5d361de39d97 (DEFAULT_TABLE table=test-table [id=50f29e847c5d4c3aa845f2fe9bd7bf4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:12:27.656101 16014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd61772f63324aa38d5a5d361de39d97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:12:27.656010 15880 tablet_service.cc:1467] Processing CreateTablet for tablet fd61772f63324aa38d5a5d361de39d97 (DEFAULT_TABLE table=test-table [id=50f29e847c5d4c3aa845f2fe9bd7bf4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:12:27.656235 16223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:27.650778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45897 (local address 127.0.0.1:51346)
0924 07:12:27.650899 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:27.650943 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:27.651011 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:27.651516 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:27.651517 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:27.651519 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:12:27.651594 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:27.651597 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:27.652907 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:27.652908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:27.653984 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:27.653986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:27.656034 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:27.656035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:27.656096 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:27.656101 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:12:27.656132 (+ 31us) client_negotiation.cc:240] Negotiation successful
0924 07:12:27.656170 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":5,"thread_start_us":58,"threads_started":1}
I20260924 07:12:27.656360 15880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd61772f63324aa38d5a5d361de39d97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:12:27.656601 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:27.650878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51346 (local address 127.0.98.3:45897)
0924 07:12:27.651358 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:27.651364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:27.651372 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:12:27.651404 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:27.651406 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:27.651419 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:27.651667 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:12:27.651738 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:27.652804 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:27.654848 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:27.655930 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:27.656406 (+ 476us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:27.656463 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:27.656490 (+ 27us) server_negotiation.cc:299] Negotiation successful
0924 07:12:27.656530 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":87,"threads_started":1}
I20260924 07:12:27.658957 16228 tablet_bootstrap.cc:492] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1: Bootstrap starting.
I20260924 07:12:27.659121 16229 tablet_bootstrap.cc:492] T fd61772f63324aa38d5a5d361de39d97 P 54431bf7bec5412483921836103ad70d: Bootstrap starting.
I20260924 07:12:27.659370 16148 tablet_service.cc:1467] Processing CreateTablet for tablet fd61772f63324aa38d5a5d361de39d97 (DEFAULT_TABLE table=test-table [id=50f29e847c5d4c3aa845f2fe9bd7bf4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:12:27.659686 16148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd61772f63324aa38d5a5d361de39d97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:12:27.659857 16228 tablet_bootstrap.cc:654] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1: Neither blocks nor log segments found. Creating new log.
I20260924 07:12:27.659862 16229 tablet_bootstrap.cc:654] T fd61772f63324aa38d5a5d361de39d97 P 54431bf7bec5412483921836103ad70d: Neither blocks nor log segments found. Creating new log.
I20260924 07:12:27.660307 16229 log.cc:824] T fd61772f63324aa38d5a5d361de39d97 P 54431bf7bec5412483921836103ad70d: Log is configured to *not* fsync() on all Append() calls
I20260924 07:12:27.660409 16228 log.cc:824] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1: Log is configured to *not* fsync() on all Append() calls
I20260924 07:12:27.661020 16229 tablet_bootstrap.cc:492] T fd61772f63324aa38d5a5d361de39d97 P 54431bf7bec5412483921836103ad70d: No bootstrap required, opened a new log
I20260924 07:12:27.661074 16229 ts_tablet_manager.cc:1397] T fd61772f63324aa38d5a5d361de39d97 P 54431bf7bec5412483921836103ad70d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:12:27.662413 16232 tablet_bootstrap.cc:492] T fd61772f63324aa38d5a5d361de39d97 P 5ad5251214014400b3fb8f34d80c0363: Bootstrap starting.
I20260924 07:12:27.662559 16229 raft_consensus.cc:348] T fd61772f63324aa38d5a5d361de39d97 P 54431bf7bec5412483921836103ad70d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad5251214014400b3fb8f34d80c0363" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45897 } } peers { permanent_uuid: "67d8d14af05940f0b8408e3b8162d1a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35475 } } peers { permanent_uuid: "54431bf7bec5412483921836103ad70d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43979 } }
I20260924 07:12:27.662727 16229 raft_consensus.cc:374] T fd61772f63324aa38d5a5d361de39d97 P 54431bf7bec5412483921836103ad70d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:12:27.662751 16229 raft_consensus.cc:729] T fd61772f63324aa38d5a5d361de39d97 P 54431bf7bec5412483921836103ad70d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54431bf7bec5412483921836103ad70d, State: Initialized, Role: FOLLOWER
I20260924 07:12:27.662883 16229 consensus_queue.cc:260] T fd61772f63324aa38d5a5d361de39d97 P 54431bf7bec5412483921836103ad70d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad5251214014400b3fb8f34d80c0363" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45897 } } peers { permanent_uuid: "67d8d14af05940f0b8408e3b8162d1a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35475 } } peers { permanent_uuid: "54431bf7bec5412483921836103ad70d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43979 } }
I20260924 07:12:27.663039 16232 tablet_bootstrap.cc:654] T fd61772f63324aa38d5a5d361de39d97 P 5ad5251214014400b3fb8f34d80c0363: Neither blocks nor log segments found. Creating new log.
I20260924 07:12:27.663196 16229 ts_tablet_manager.cc:1428] T fd61772f63324aa38d5a5d361de39d97 P 54431bf7bec5412483921836103ad70d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 07:12:27.663313 15946 heartbeater.cc:503] Master 127.0.98.62:37127 was elected leader, sending a full tablet report...
I20260924 07:12:27.663594 16232 log.cc:824] T fd61772f63324aa38d5a5d361de39d97 P 5ad5251214014400b3fb8f34d80c0363: Log is configured to *not* fsync() on all Append() calls
I20260924 07:12:27.664024 16228 tablet_bootstrap.cc:492] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1: No bootstrap required, opened a new log
I20260924 07:12:27.664090 16228 ts_tablet_manager.cc:1397] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260924 07:12:27.664331 16232 tablet_bootstrap.cc:492] T fd61772f63324aa38d5a5d361de39d97 P 5ad5251214014400b3fb8f34d80c0363: No bootstrap required, opened a new log
I20260924 07:12:27.664459 16232 ts_tablet_manager.cc:1397] T fd61772f63324aa38d5a5d361de39d97 P 5ad5251214014400b3fb8f34d80c0363: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:12:27.665582 16228 raft_consensus.cc:348] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad5251214014400b3fb8f34d80c0363" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45897 } } peers { permanent_uuid: "67d8d14af05940f0b8408e3b8162d1a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35475 } } peers { permanent_uuid: "54431bf7bec5412483921836103ad70d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43979 } }
I20260924 07:12:27.665711 16228 raft_consensus.cc:374] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:12:27.665735 16228 raft_consensus.cc:729] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67d8d14af05940f0b8408e3b8162d1a1, State: Initialized, Role: FOLLOWER
I20260924 07:12:27.665843 16228 consensus_queue.cc:260] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad5251214014400b3fb8f34d80c0363" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45897 } } peers { permanent_uuid: "67d8d14af05940f0b8408e3b8162d1a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35475 } } peers { permanent_uuid: "54431bf7bec5412483921836103ad70d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43979 } }
I20260924 07:12:27.665874 16232 raft_consensus.cc:348] T fd61772f63324aa38d5a5d361de39d97 P 5ad5251214014400b3fb8f34d80c0363 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad5251214014400b3fb8f34d80c0363" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45897 } } peers { permanent_uuid: "67d8d14af05940f0b8408e3b8162d1a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35475 } } peers { permanent_uuid: "54431bf7bec5412483921836103ad70d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43979 } }
I20260924 07:12:27.666074 16232 raft_consensus.cc:374] T fd61772f63324aa38d5a5d361de39d97 P 5ad5251214014400b3fb8f34d80c0363 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:12:27.666112 16228 ts_tablet_manager.cc:1428] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:12:27.666153 16080 heartbeater.cc:503] Master 127.0.98.62:37127 was elected leader, sending a full tablet report...
I20260924 07:12:27.666167 16232 raft_consensus.cc:729] T fd61772f63324aa38d5a5d361de39d97 P 5ad5251214014400b3fb8f34d80c0363 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ad5251214014400b3fb8f34d80c0363, State: Initialized, Role: FOLLOWER
I20260924 07:12:27.666330 16232 consensus_queue.cc:260] T fd61772f63324aa38d5a5d361de39d97 P 5ad5251214014400b3fb8f34d80c0363 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad5251214014400b3fb8f34d80c0363" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45897 } } peers { permanent_uuid: "67d8d14af05940f0b8408e3b8162d1a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35475 } } peers { permanent_uuid: "54431bf7bec5412483921836103ad70d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43979 } }
I20260924 07:12:27.666594 16232 ts_tablet_manager.cc:1428] T fd61772f63324aa38d5a5d361de39d97 P 5ad5251214014400b3fb8f34d80c0363: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:12:27.666610 16214 heartbeater.cc:503] Master 127.0.98.62:37127 was elected leader, sending a full tablet report...
W20260924 07:12:27.728855 16081 tablet.cc:2369] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:12:27.828460 15947 tablet.cc:2369] T fd61772f63324aa38d5a5d361de39d97 P 54431bf7bec5412483921836103ad70d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:12:27.872196 16215 tablet.cc:2369] T fd61772f63324aa38d5a5d361de39d97 P 5ad5251214014400b3fb8f34d80c0363: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:12:27.962910 16235 raft_consensus.cc:482] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:12:27.963032 16235 raft_consensus.cc:504] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad5251214014400b3fb8f34d80c0363" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45897 } } peers { permanent_uuid: "67d8d14af05940f0b8408e3b8162d1a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35475 } } peers { permanent_uuid: "54431bf7bec5412483921836103ad70d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43979 } }
I20260924 07:12:27.963349 16235 leader_election.cc:288] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ad5251214014400b3fb8f34d80c0363 (127.0.98.3:45897), 54431bf7bec5412483921836103ad70d (127.0.98.1:43979)
I20260924 07:12:27.966602 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:27.963758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60013 (local address 127.0.98.3:45897)
0924 07:12:27.963804 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:27.963806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:27.963815 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:12:27.963865 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:27.963866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:27.963880 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:27.963931 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:12:27.964143 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:27.964631 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:27.965886 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:27.966258 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:27.966345 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:27.966356 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:27.966502 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 07:12:27.966544 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 07:12:27.966600 16082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:27.963547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45897 (local address 127.0.98.2:60013)
0924 07:12:27.963605 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:27.963618 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:27.963665 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:27.963987 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:27.963988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:27.963990 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:12:27.964069 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:27.964070 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:27.964721 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:27.964722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:27.965795 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:27.965798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:27.966335 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:27.966336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:27.966420 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:27.966423 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:12:27.966484 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 07:12:27.966527 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 07:12:27.966940 16168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd61772f63324aa38d5a5d361de39d97" candidate_uuid: "67d8d14af05940f0b8408e3b8162d1a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ad5251214014400b3fb8f34d80c0363" is_pre_election: true
I20260924 07:12:27.967108 16168 raft_consensus.cc:2454] T fd61772f63324aa38d5a5d361de39d97 P 5ad5251214014400b3fb8f34d80c0363 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67d8d14af05940f0b8408e3b8162d1a1 in term 0.
I20260924 07:12:27.967134 16237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:27.963676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43979 (local address 127.0.98.2:38679)
0924 07:12:27.963782 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:27.963793 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:27.963874 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:27.964002 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:27.964003 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:27.964004 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:12:27.964067 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:27.964069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:27.965561 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:27.965561 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:27.966293 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:27.966294 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:27.966887 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:27.966888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:27.966961 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:27.966966 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:12:27.967025 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 07:12:27.967063 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":59,"threads_started":1}
I20260924 07:12:27.967196 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:27.963766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38679 (local address 127.0.98.1:43979)
0924 07:12:27.963820 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:27.963822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:27.963880 (+ 58us) server_negotiation.cc:407] Connection header received
0924 07:12:27.963928 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:27.963929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:27.963942 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:27.963991 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:12:27.964771 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:27.965474 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:27.966383 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:27.966819 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:27.966884 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:27.966892 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:27.967036 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 07:12:27.967100 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 07:12:27.967331 15967 leader_election.cc:302] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ad5251214014400b3fb8f34d80c0363, 67d8d14af05940f0b8408e3b8162d1a1; no voters:
I20260924 07:12:27.967475 16235 raft_consensus.cc:2781] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:12:27.967512 15900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd61772f63324aa38d5a5d361de39d97" candidate_uuid: "67d8d14af05940f0b8408e3b8162d1a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54431bf7bec5412483921836103ad70d" is_pre_election: true
I20260924 07:12:27.967562 16235 raft_consensus.cc:482] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:12:27.967603 16235 raft_consensus.cc:3037] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:12:27.967644 15900 raft_consensus.cc:2454] T fd61772f63324aa38d5a5d361de39d97 P 54431bf7bec5412483921836103ad70d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67d8d14af05940f0b8408e3b8162d1a1 in term 0.
I20260924 07:12:27.968401 16235 raft_consensus.cc:504] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad5251214014400b3fb8f34d80c0363" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45897 } } peers { permanent_uuid: "67d8d14af05940f0b8408e3b8162d1a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35475 } } peers { permanent_uuid: "54431bf7bec5412483921836103ad70d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43979 } }
I20260924 07:12:27.968556 16235 leader_election.cc:288] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1 [CANDIDATE]: Term 1 election: Requested vote from peers 5ad5251214014400b3fb8f34d80c0363 (127.0.98.3:45897), 54431bf7bec5412483921836103ad70d (127.0.98.1:43979)
I20260924 07:12:27.968763 16168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd61772f63324aa38d5a5d361de39d97" candidate_uuid: "67d8d14af05940f0b8408e3b8162d1a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ad5251214014400b3fb8f34d80c0363"
I20260924 07:12:27.968773 15900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd61772f63324aa38d5a5d361de39d97" candidate_uuid: "67d8d14af05940f0b8408e3b8162d1a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54431bf7bec5412483921836103ad70d"
I20260924 07:12:27.968848 16168 raft_consensus.cc:3037] T fd61772f63324aa38d5a5d361de39d97 P 5ad5251214014400b3fb8f34d80c0363 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:12:27.968868 15900 raft_consensus.cc:3037] T fd61772f63324aa38d5a5d361de39d97 P 54431bf7bec5412483921836103ad70d [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:12:27.969566 15900 raft_consensus.cc:2454] T fd61772f63324aa38d5a5d361de39d97 P 54431bf7bec5412483921836103ad70d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67d8d14af05940f0b8408e3b8162d1a1 in term 1.
I20260924 07:12:27.969653 16168 raft_consensus.cc:2454] T fd61772f63324aa38d5a5d361de39d97 P 5ad5251214014400b3fb8f34d80c0363 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67d8d14af05940f0b8408e3b8162d1a1 in term 1.
I20260924 07:12:27.969715 15967 leader_election.cc:302] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54431bf7bec5412483921836103ad70d, 67d8d14af05940f0b8408e3b8162d1a1; no voters:
I20260924 07:12:27.969802 16235 raft_consensus.cc:2781] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:12:27.969973 16235 raft_consensus.cc:686] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1 [term 1 LEADER]: Becoming Leader. State: Replica: 67d8d14af05940f0b8408e3b8162d1a1, State: Running, Role: LEADER
I20260924 07:12:27.970067 16235 consensus_queue.cc:237] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad5251214014400b3fb8f34d80c0363" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45897 } } peers { permanent_uuid: "67d8d14af05940f0b8408e3b8162d1a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35475 } } peers { permanent_uuid: "54431bf7bec5412483921836103ad70d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43979 } }
I20260924 07:12:27.970917 15755 catalog_manager.cc:5495] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67d8d14af05940f0b8408e3b8162d1a1 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "67d8d14af05940f0b8408e3b8162d1a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad5251214014400b3fb8f34d80c0363" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67d8d14af05940f0b8408e3b8162d1a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35475 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54431bf7bec5412483921836103ad70d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43979 } health_report { overall_health: UNKNOWN } } }
I20260924 07:12:28.006651 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:28.004305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45880 (local address 127.0.98.2:35475)
0924 07:12:28.004372 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:28.004374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:28.004383 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:12:28.004452 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:28.004453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:28.004471 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:28.004523 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:12:28.004651 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:28.005156 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:28.005878 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:28.006051 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:28.006129 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:28.006295 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:12:28.006405 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:12:28.006465 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:28.006503 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 07:12:28.006520 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:12:28.008464 15900 raft_consensus.cc:1261] T fd61772f63324aa38d5a5d361de39d97 P 54431bf7bec5412483921836103ad70d [term 1 FOLLOWER]: Refusing update from remote peer 67d8d14af05940f0b8408e3b8162d1a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:12:28.008484 16168 raft_consensus.cc:1261] T fd61772f63324aa38d5a5d361de39d97 P 5ad5251214014400b3fb8f34d80c0363 [term 1 FOLLOWER]: Refusing update from remote peer 67d8d14af05940f0b8408e3b8162d1a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:12:28.008858 16235 consensus_queue.cc:1035] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54431bf7bec5412483921836103ad70d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:12:28.009006 16238 consensus_queue.cc:1035] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ad5251214014400b3fb8f34d80c0363" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:12:28.011149 16241 mvcc.cc:205] Tried to move back new op lower bound from 7332798251039944704 to 7332798250886103040. Current Snapshot: MvccSnapshot[applied={T|T < 7332798251039944704}]
I20260924 07:12:28.011444 16243 mvcc.cc:205] Tried to move back new op lower bound from 7332798251039944704 to 7332798250886103040. Current Snapshot: MvccSnapshot[applied={T|T < 7332798251039944704}]
I20260924 07:12:28.011921 16242 mvcc.cc:205] Tried to move back new op lower bound from 7332798251039944704 to 7332798250886103040. Current Snapshot: MvccSnapshot[applied={T|T < 7332798251039944704 or (T in {7332798251039944704})}]
I20260924 07:12:28.013542 15754 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:12:28.013644 15754 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:12:28.014417 15754 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc: Sending DeleteTablet for 3 replicas of tablet fd61772f63324aa38d5a5d361de39d97
I20260924 07:12:28.014750 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 15817
I20260924 07:12:28.014748 16148 tablet_service.cc:1514] Processing DeleteTablet for tablet fd61772f63324aa38d5a5d361de39d97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:12:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51346
I20260924 07:12:28.014743 16014 tablet_service.cc:1514] Processing DeleteTablet for tablet fd61772f63324aa38d5a5d361de39d97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:12:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45876
W20260924 07:12:28.019433 15739 connection.cc:535] client connection to 127.0.98.1:43979 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 07:12:28.019526 15739 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 07:12:28.019577 16251 tablet_replica.cc:330] T fd61772f63324aa38d5a5d361de39d97 P 5ad5251214014400b3fb8f34d80c0363: stopping tablet replica
I20260924 07:12:28.019656 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 15951
I20260924 07:12:28.019681 16251 raft_consensus.cc:2229] T fd61772f63324aa38d5a5d361de39d97 P 5ad5251214014400b3fb8f34d80c0363 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:12:28.019781 16251 raft_consensus.cc:2258] T fd61772f63324aa38d5a5d361de39d97 P 5ad5251214014400b3fb8f34d80c0363 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:12:28.020218 16251 ts_tablet_manager.cc:1905] T fd61772f63324aa38d5a5d361de39d97 P 5ad5251214014400b3fb8f34d80c0363: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:12:28.020767 16223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:28.020612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43979 (local address 127.0.0.1:36810)
0924 07:12:28.020666 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:28.020699 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 07:12:28.020920 15739 catalog_manager.cc:4584] TS 54431bf7bec5412483921836103ad70d (127.0.98.1:43979): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fd61772f63324aa38d5a5d361de39d97: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43979: connect: Connection refused (error 111)
I20260924 07:12:28.022087 16251 ts_tablet_manager.cc:1918] T fd61772f63324aa38d5a5d361de39d97 P 5ad5251214014400b3fb8f34d80c0363: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:12:28.022176 16251 log.cc:1196] T fd61772f63324aa38d5a5d361de39d97 P 5ad5251214014400b3fb8f34d80c0363: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/wal/wals/fd61772f63324aa38d5a5d361de39d97
I20260924 07:12:28.022490 16251 ts_tablet_manager.cc:1939] T fd61772f63324aa38d5a5d361de39d97 P 5ad5251214014400b3fb8f34d80c0363: Deleting consensus metadata
I20260924 07:12:28.022891 15739 catalog_manager.cc:4841] TS 5ad5251214014400b3fb8f34d80c0363 (127.0.98.3:45897): tablet fd61772f63324aa38d5a5d361de39d97 (table test-table [id=50f29e847c5d4c3aa845f2fe9bd7bf4d]) successfully deleted
I20260924 07:12:28.025447 16223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:28.025340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35475 (local address 127.0.0.1:45888)
0924 07:12:28.025387 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:28.025416 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:35475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 07:12:28.025456 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 16085
I20260924 07:12:28.030690 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 15723
I20260924 07:12:48.036039 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37127
--webserver_interface=127.0.98.62
--webserver_port=41239
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:12:48.116659 16254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:48.116904 16254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:48.116940 16254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:48.118060 16254 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 07:12:48.118126 16254 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 07:12:48.118148 16254 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 07:12:48.118167 16254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:12:48.118197 16254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:48.118222 16254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:48.118250 16254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:12:48.118276 16254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:12:48.119825 16254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:12:48.120092 16254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:48.120311 16254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:48.129451 16260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:48.129442 16254 server_base.cc:1034] running on GCE node
W20260924 07:12:48.129444 16261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:48.129568 16263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:48.129966 16254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:48.130306 16254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:48.131474 16254 hybrid_clock.cc:648] HybridClock initialized: now 1790233968131457 us; error 35 us; skew 500 ppm
Sep 24 07:12:48 dist-test-slave-14ww krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:12:48.134056 16254 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:12:48.134523 16254 webserver.cc:460] Webserver started at http://127.0.98.62:41239/ using document root <none> and password file <none>
I20260924 07:12:48.134776 16254 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:48.134856 16254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:48.136310 16254 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:12:48.137116 16270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:48.137586 16254 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 07:12:48.137693 16254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "fc6625c0c7d047e594a80ff678e26bdc"
format_stamp: "Formatted at 2026-09-24 07:12:27 on dist-test-slave-14ww"
server_key: "782ee73060c760169f463f8057bac125"
server_key_iv: "086b8c1a314427f9c102325a87a593d4"
server_key_version: "encryptionkey@0"
I20260924 07:12:48.138084 16254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:48.164815 16254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:48.166942 16254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:48.167116 16254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:48.171449 16254 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37127
I20260924 07:12:48.171509 16322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37127 every 8 connection(s)
I20260924 07:12:48.171916 16254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:12:48.173473 16323 sys_catalog.cc:263] Verifying existing consensus state
I20260924 07:12:48.174155 16323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc: Bootstrap starting.
I20260924 07:12:48.181470 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 16254
I20260924 07:12:48.181782 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43979
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37127
--webserver_port=36427
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:12:48.182901 16323 log.cc:824] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc: Log is configured to *not* fsync() on all Append() calls
I20260924 07:12:48.185127 16323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:12:48.185425 16323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc: Bootstrap complete.
I20260924 07:12:48.187813 16323 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6625c0c7d047e594a80ff678e26bdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37127 } }
I20260924 07:12:48.188089 16323 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc6625c0c7d047e594a80ff678e26bdc, State: Initialized, Role: FOLLOWER
I20260924 07:12:48.188287 16323 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6625c0c7d047e594a80ff678e26bdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37127 } }
I20260924 07:12:48.188369 16323 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:12:48.188429 16323 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:12:48.188486 16323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [term 1 FOLLOWER]: Advancing to term 2
I20260924 07:12:48.189615 16323 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6625c0c7d047e594a80ff678e26bdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37127 } }
I20260924 07:12:48.189741 16323 leader_election.cc:302] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc6625c0c7d047e594a80ff678e26bdc; no voters:
I20260924 07:12:48.189963 16323 leader_election.cc:288] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 07:12:48.190004 16327 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [term 2 FOLLOWER]: Leader election won for term 2
I20260924 07:12:48.190125 16327 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [term 2 LEADER]: Becoming Leader. State: Replica: fc6625c0c7d047e594a80ff678e26bdc, State: Running, Role: LEADER
I20260924 07:12:48.190248 16323 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:12:48.190274 16327 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6625c0c7d047e594a80ff678e26bdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37127 } }
I20260924 07:12:48.190558 16327 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc6625c0c7d047e594a80ff678e26bdc. Latest consensus state: current_term: 2 leader_uuid: "fc6625c0c7d047e594a80ff678e26bdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6625c0c7d047e594a80ff678e26bdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37127 } } }
I20260924 07:12:48.190627 16327 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [sys.catalog]: This master's current role is: LEADER
I20260924 07:12:48.190595 16328 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fc6625c0c7d047e594a80ff678e26bdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6625c0c7d047e594a80ff678e26bdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37127 } } }
I20260924 07:12:48.190831 16332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:12:48.190845 16328 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [sys.catalog]: This master's current role is: LEADER
I20260924 07:12:48.191766 16332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:12:48.192135 16332 catalog_manager.cc:1259] Loaded cluster ID: 206e4d8a723e4869b8c0297eab5c66d0
I20260924 07:12:48.192219 16332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:12:48.193114 16332 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:12:48.193295 16332 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc: Loaded TSK: 0
I20260924 07:12:48.205499 16332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc: Generated new TSK 1
I20260924 07:12:48.205610 16332 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 07:12:48.281419 16325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:48.281661 16325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:48.281701 16325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:48.282842 16325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:48.282909 16325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:48.282954 16325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:12:48.284407 16325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43979
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36427
--tserver_master_addrs=127.0.98.62:37127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:12:48.284678 16325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:48.284936 16325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:48.290809 16351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:48.290853 16353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:48.290899 16350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:48.291587 16325 server_base.cc:1034] running on GCE node
I20260924 07:12:48.291801 16325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:48.292042 16325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:48.293210 16325 hybrid_clock.cc:648] HybridClock initialized: now 1790233968293195 us; error 31 us; skew 500 ppm
Sep 24 07:12:48 dist-test-slave-14ww krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:12:48.295674 16325 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:12:48.296140 16325 webserver.cc:460] Webserver started at http://127.0.98.1:36427/ using document root <none> and password file <none>
I20260924 07:12:48.296429 16325 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:48.296525 16325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:48.297987 16325 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:12:48.298703 16360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:48.298852 16325 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:12:48.298954 16325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/wal
uuid: "54431bf7bec5412483921836103ad70d"
format_stamp: "Formatted at 2026-09-24 07:12:27 on dist-test-slave-14ww"
server_key: "49aeb66134fa49a1842c4c882ef8d5a7"
server_key_iv: "f4d1f6c520ee3b1b71fc447996c32afd"
server_key_version: "encryptionkey@0"
I20260924 07:12:48.299309 16325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:48.308965 16325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:48.311223 16325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:48.311368 16325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:48.311602 16325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:12:48.312036 16367 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 07:12:48.314023 16325 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 07:12:48.314108 16325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 07:12:48.314159 16325 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 07:12:48.314797 16325 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 07:12:48.314883 16325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 07:12:48.314919 16367 tablet_bootstrap.cc:492] T fd61772f63324aa38d5a5d361de39d97 P 54431bf7bec5412483921836103ad70d: Bootstrap starting.
I20260924 07:12:48.322256 16325 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43979
I20260924 07:12:48.322620 16325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:12:48.324921 16474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43979 every 8 connection(s)
I20260924 07:12:48.325999 16367 log.cc:824] T fd61772f63324aa38d5a5d361de39d97 P 54431bf7bec5412483921836103ad70d: Log is configured to *not* fsync() on all Append() calls
I20260924 07:12:48.326913 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 16325
I20260924 07:12:48.327194 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35475
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37127
--webserver_port=45559
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:12:48.328872 16367 tablet_bootstrap.cc:492] T fd61772f63324aa38d5a5d361de39d97 P 54431bf7bec5412483921836103ad70d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:12:48.329123 16367 tablet_bootstrap.cc:492] T fd61772f63324aa38d5a5d361de39d97 P 54431bf7bec5412483921836103ad70d: Bootstrap complete.
I20260924 07:12:48.330328 16367 ts_tablet_manager.cc:1397] T fd61772f63324aa38d5a5d361de39d97 P 54431bf7bec5412483921836103ad70d: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.003s
I20260924 07:12:48.331700 16367 raft_consensus.cc:348] T fd61772f63324aa38d5a5d361de39d97 P 54431bf7bec5412483921836103ad70d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad5251214014400b3fb8f34d80c0363" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45897 } } peers { permanent_uuid: "67d8d14af05940f0b8408e3b8162d1a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35475 } } peers { permanent_uuid: "54431bf7bec5412483921836103ad70d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43979 } }
I20260924 07:12:48.332042 16367 raft_consensus.cc:729] T fd61772f63324aa38d5a5d361de39d97 P 54431bf7bec5412483921836103ad70d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54431bf7bec5412483921836103ad70d, State: Initialized, Role: FOLLOWER
I20260924 07:12:48.332188 16367 consensus_queue.cc:260] T fd61772f63324aa38d5a5d361de39d97 P 54431bf7bec5412483921836103ad70d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad5251214014400b3fb8f34d80c0363" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45897 } } peers { permanent_uuid: "67d8d14af05940f0b8408e3b8162d1a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35475 } } peers { permanent_uuid: "54431bf7bec5412483921836103ad70d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43979 } }
I20260924 07:12:48.332461 16367 ts_tablet_manager.cc:1428] T fd61772f63324aa38d5a5d361de39d97 P 54431bf7bec5412483921836103ad70d: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
Sep 24 07:12:48 dist-test-slave-14ww krb5kdc[15710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:12:48.337720 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:48.323331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.1:43637)
0924 07:12:48.323524 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:48.323568 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:48.324069 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:48.327047 (+ 2978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:48.327051 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:48.327415 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:48.327666 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:48.327673 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:48.329402 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:48.329403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:48.330798 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:48.330800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:48.331638 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:48.331639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:48.331779 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:48.332635 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:48.332643 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:48.334105 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:12:48.336107 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:48.336115 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:48.336117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:48.336350 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:48.336619 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:48.336620 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:48.336620 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:48.336767 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:48.337116 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:12:48.337122 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:12:48.337255 (+ 133us) client_negotiation.cc:769] Sending connection context
0924 07:12:48.337357 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 07:12:48.337525 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":121,"threads_started":1}
I20260924 07:12:48.337738 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:48.323625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43637 (local address 127.0.98.62:37127)
0924 07:12:48.326367 (+ 2742us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:48.326375 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:48.326428 (+ 53us) server_negotiation.cc:407] Connection header received
0924 07:12:48.326557 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:48.326564 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:48.326617 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:48.326772 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:48.327804 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:48.329255 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:48.330902 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:48.331529 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:48.332883 (+ 1354us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:48.334251 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:48.334256 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:48.334257 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:48.334271 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:48.335944 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:48.336452 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:48.336455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:48.336455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:48.336536 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:48.336826 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:48.336827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:48.336827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:48.337037 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:48.337136 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:48.337429 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 07:12:48.337549 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2692,"thread_start_us":75,"threads_started":1}
I20260924 07:12:48.338692 16475 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37127
I20260924 07:12:48.338783 16475 heartbeater.cc:464] Registering TS with master...
I20260924 07:12:48.338956 16475 heartbeater.cc:511] Master 127.0.98.62:37127 requested a full tablet report, sending...
I20260924 07:12:48.339668 16287 ts_manager.cc:194] Registered new tserver with Master: 54431bf7bec5412483921836103ad70d (127.0.98.1:43979)
I20260924 07:12:48.340847 16287 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43637
I20260924 07:12:48.341722 16475 heartbeater.cc:503] Master 127.0.98.62:37127 was elected leader, sending a full tablet report...
I20260924 07:12:48.347891 16483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:48.340450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36672 (local address 127.0.98.1:43979)
0924 07:12:48.344858 (+ 4408us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:48.344863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:48.344874 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:12:48.344912 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:48.344914 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:48.344929 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:48.345411 (+ 482us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:12:48.345447 (+ 36us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:48.346048 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:48.347021 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:48.347485 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:48.347571 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:48.347626 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:48.347789 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 07:12:48.347826 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4377,"thread_start_us":77,"threads_started":1}
I20260924 07:12:48.347903 16482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:48.340225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43979 (local address 127.0.0.1:36672)
0924 07:12:48.340814 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:48.340847 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:48.341010 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:48.345214 (+ 4204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:48.345217 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:48.345219 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:12:48.345286 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:48.345288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:48.346134 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:48.346135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:48.346930 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:48.346932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:48.347583 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:48.347583 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:48.347665 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:48.347684 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 07:12:48.347771 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 07:12:48.347824 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":87,"threads_started":1}
I20260924 07:12:48.348511 16409 tablet_service.cc:1514] Processing DeleteTablet for tablet fd61772f63324aa38d5a5d361de39d97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:12:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36672
I20260924 07:12:48.349047 16484 tablet_replica.cc:330] stopping tablet replica
I20260924 07:12:48.349169 16484 raft_consensus.cc:2229] T fd61772f63324aa38d5a5d361de39d97 P 54431bf7bec5412483921836103ad70d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:12:48.349237 16484 raft_consensus.cc:2258] T fd61772f63324aa38d5a5d361de39d97 P 54431bf7bec5412483921836103ad70d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:12:48.349530 16484 ts_tablet_manager.cc:1905] T fd61772f63324aa38d5a5d361de39d97 P 54431bf7bec5412483921836103ad70d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:12:48.351665 16484 ts_tablet_manager.cc:1918] T fd61772f63324aa38d5a5d361de39d97 P 54431bf7bec5412483921836103ad70d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:12:48.351760 16484 log.cc:1196] T fd61772f63324aa38d5a5d361de39d97 P 54431bf7bec5412483921836103ad70d: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/wal/wals/fd61772f63324aa38d5a5d361de39d97
I20260924 07:12:48.352023 16484 ts_tablet_manager.cc:1939] T fd61772f63324aa38d5a5d361de39d97 P 54431bf7bec5412483921836103ad70d: Deleting consensus metadata
I20260924 07:12:48.352499 16271 catalog_manager.cc:4841] TS 54431bf7bec5412483921836103ad70d (127.0.98.1:43979): tablet fd61772f63324aa38d5a5d361de39d97 (table test-table [id=50f29e847c5d4c3aa845f2fe9bd7bf4d]) successfully deleted
W20260924 07:12:48.416800 16480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:48.417001 16480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:48.417026 16480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:48.418238 16480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:48.418280 16480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:48.418308 16480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:12:48.419684 16480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35475
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45559
--tserver_master_addrs=127.0.98.62:37127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:12:48.419857 16480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:48.420019 16480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:12:48.425028 16480 server_base.cc:1034] running on GCE node
W20260924 07:12:48.424975 16492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:48.425030 16489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:48.424996 16490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:48.425437 16480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:48.425689 16480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:48.426855 16480 hybrid_clock.cc:648] HybridClock initialized: now 1790233968426845 us; error 37 us; skew 500 ppm
Sep 24 07:12:48 dist-test-slave-14ww krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:12:48.429406 16480 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:12:48.429811 16480 webserver.cc:460] Webserver started at http://127.0.98.2:45559/ using document root <none> and password file <none>
I20260924 07:12:48.430030 16480 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:48.430110 16480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:48.431399 16480 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:12:48.432077 16499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:48.432250 16480 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 07:12:48.432351 16480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/wal
uuid: "67d8d14af05940f0b8408e3b8162d1a1"
format_stamp: "Formatted at 2026-09-24 07:12:27 on dist-test-slave-14ww"
server_key: "d33af9cf35d87548fe4aa025c70e1d79"
server_key_iv: "5c5a48d2cdde09b55f42b9ef12d62fb3"
server_key_version: "encryptionkey@0"
I20260924 07:12:48.432674 16480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:48.449486 16480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:48.451713 16480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:48.451864 16480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:48.452071 16480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:12:48.452472 16506 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 07:12:48.453971 16480 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 07:12:48.454039 16480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 07:12:48.454082 16480 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 07:12:48.454679 16480 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 07:12:48.454731 16480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 07:12:48.454779 16506 tablet_bootstrap.cc:492] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1: Bootstrap starting.
I20260924 07:12:48.461607 16480 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35475
I20260924 07:12:48.461678 16613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35475 every 8 connection(s)
I20260924 07:12:48.461977 16480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:12:48.463497 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 16480
I20260924 07:12:48.463795 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45897
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37127
--webserver_port=33445
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:12:48.465736 16506 log.cc:824] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1: Log is configured to *not* fsync() on all Append() calls
I20260924 07:12:48.468552 16506 tablet_bootstrap.cc:492] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
Sep 24 07:12:48 dist-test-slave-14ww krb5kdc[15710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:12:48.468916 16506 tablet_bootstrap.cc:492] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1: Bootstrap complete.
I20260924 07:12:48.470161 16506 ts_tablet_manager.cc:1397] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.006s
I20260924 07:12:48.472136 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:48.462716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42743 (local address 127.0.98.62:37127)
0924 07:12:48.462789 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:48.462791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:48.463023 (+ 232us) server_negotiation.cc:407] Connection header received
0924 07:12:48.463830 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:48.463831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:48.463848 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:48.463913 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:48.464851 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:48.465480 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:48.466709 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:48.466929 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:48.467023 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:48.469290 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:48.469292 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:48.469292 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:48.469301 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:48.470756 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:48.471182 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:48.471184 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:48.471184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:48.471244 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:48.471496 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:48.471497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:48.471498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:48.471612 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:48.471689 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:48.471952 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 07:12:48.472044 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 07:12:48.471781 16506 raft_consensus.cc:348] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad5251214014400b3fb8f34d80c0363" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45897 } } peers { permanent_uuid: "67d8d14af05940f0b8408e3b8162d1a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35475 } } peers { permanent_uuid: "54431bf7bec5412483921836103ad70d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43979 } }
I20260924 07:12:48.472180 16506 raft_consensus.cc:729] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67d8d14af05940f0b8408e3b8162d1a1, State: Initialized, Role: FOLLOWER
I20260924 07:12:48.472374 16616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:48.462655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.2:42743)
0924 07:12:48.462863 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:48.462918 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:48.463683 (+ 765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:48.464009 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:48.464014 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:48.464413 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:48.464753 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:48.464763 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:48.465598 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:48.465599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:48.466617 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:48.466620 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:48.467051 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:48.467053 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:48.467169 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:48.467702 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:48.467710 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:48.469196 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:12:48.470860 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:48.470866 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:48.470868 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:48.471097 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:48.471312 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:48.471312 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:48.471313 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:48.471425 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:48.471680 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:12:48.471686 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:12:48.471822 (+ 136us) client_negotiation.cc:769] Sending connection context
0924 07:12:48.471932 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 07:12:48.472097 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":109,"threads_started":1}
I20260924 07:12:48.472350 16506 consensus_queue.cc:260] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad5251214014400b3fb8f34d80c0363" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45897 } } peers { permanent_uuid: "67d8d14af05940f0b8408e3b8162d1a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35475 } } peers { permanent_uuid: "54431bf7bec5412483921836103ad70d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43979 } }
I20260924 07:12:48.472628 16506 ts_tablet_manager.cc:1428] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
I20260924 07:12:48.472941 16614 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37127
I20260924 07:12:48.473053 16614 heartbeater.cc:464] Registering TS with master...
I20260924 07:12:48.473249 16614 heartbeater.cc:511] Master 127.0.98.62:37127 requested a full tablet report, sending...
I20260924 07:12:48.473903 16287 ts_manager.cc:194] Registered new tserver with Master: 67d8d14af05940f0b8408e3b8162d1a1 (127.0.98.2:35475)
I20260924 07:12:48.474489 16287 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42743
I20260924 07:12:48.475409 16614 heartbeater.cc:503] Master 127.0.98.62:37127 was elected leader, sending a full tablet report...
I20260924 07:12:48.487833 16482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:48.474236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35475 (local address 127.0.0.1:60562)
0924 07:12:48.474293 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:48.474304 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:48.474344 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:48.484995 (+ 10651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:48.484996 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:48.484998 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:12:48.485054 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:48.485057 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:48.485865 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:48.485866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:48.486948 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:48.486951 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:48.487587 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:48.487588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:48.487651 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:48.487654 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:12:48.487710 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 07:12:48.487756 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 07:12:48.487895 16620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:48.474308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60562 (local address 127.0.98.2:35475)
0924 07:12:48.484837 (+ 10529us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:48.484841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:48.484851 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:12:48.484878 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:48.484879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:48.484890 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:48.484982 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:12:48.485141 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:48.485782 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:48.487036 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:48.487500 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:48.487566 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:48.487602 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:48.487786 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 07:12:48.487830 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10500,"thread_start_us":77,"threads_started":1}
I20260924 07:12:48.488236 16548 tablet_service.cc:1514] Processing DeleteTablet for tablet fd61772f63324aa38d5a5d361de39d97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:12:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60562
I20260924 07:12:48.488837 16621 tablet_replica.cc:330] stopping tablet replica
I20260924 07:12:48.488946 16621 raft_consensus.cc:2229] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:12:48.489014 16621 raft_consensus.cc:2258] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:12:48.489311 16621 ts_tablet_manager.cc:1905] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:12:48.491613 16621 ts_tablet_manager.cc:1918] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:12:48.491707 16621 log.cc:1196] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/wal/wals/fd61772f63324aa38d5a5d361de39d97
I20260924 07:12:48.491940 16621 ts_tablet_manager.cc:1939] T fd61772f63324aa38d5a5d361de39d97 P 67d8d14af05940f0b8408e3b8162d1a1: Deleting consensus metadata
I20260924 07:12:48.492404 16273 catalog_manager.cc:4841] TS 67d8d14af05940f0b8408e3b8162d1a1 (127.0.98.2:35475): tablet fd61772f63324aa38d5a5d361de39d97 (table test-table [id=50f29e847c5d4c3aa845f2fe9bd7bf4d]) successfully deleted
W20260924 07:12:48.549311 16617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:12:48.549506 16617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:12:48.549530 16617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:12:48.550669 16617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:12:48.550712 16617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:12:48.550738 16617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:12:48.552152 16617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45897
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33445
--tserver_master_addrs=127.0.98.62:37127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:12:48.552383 16617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:12:48.552613 16617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:12:48.557605 16627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:48.557675 16617 server_base.cc:1034] running on GCE node
W20260924 07:12:48.557679 16626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:12:48.557679 16629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:12:48.557950 16617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:12:48.558211 16617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:12:48.559381 16617 hybrid_clock.cc:648] HybridClock initialized: now 1790233968559368 us; error 37 us; skew 500 ppm
Sep 24 07:12:48 dist-test-slave-14ww krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:12:48.561875 16617 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:12:48.562327 16617 webserver.cc:460] Webserver started at http://127.0.98.3:33445/ using document root <none> and password file <none>
I20260924 07:12:48.562546 16617 fs_manager.cc:359] Metadata directory not provided
I20260924 07:12:48.562629 16617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:12:48.564322 16617 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:12:48.565083 16636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:48.565290 16617 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:12:48.565398 16617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/wal
uuid: "5ad5251214014400b3fb8f34d80c0363"
format_stamp: "Formatted at 2026-09-24 07:12:27 on dist-test-slave-14ww"
server_key: "6d835bf92972584cf86fe74a0ff83cad"
server_key_iv: "9cbcd799cca54ea60b5af819b191b64a"
server_key_version: "encryptionkey@0"
I20260924 07:12:48.565755 16617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:12:48.580404 16617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:12:48.582973 16617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:12:48.583151 16617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:12:48.583397 16617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:12:48.583703 16617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:12:48.583755 16617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:48.583791 16617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:12:48.583817 16617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:48.590319 16617 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45897
I20260924 07:12:48.590425 16749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45897 every 8 connection(s)
I20260924 07:12:48.590785 16617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 07:12:48 dist-test-slave-14ww krb5kdc[15710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:12:48.599387 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 16617
I20260924 07:12:48.601121 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:48.591604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45993 (local address 127.0.98.62:37127)
0924 07:12:48.591652 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:48.591655 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:48.591946 (+ 291us) server_negotiation.cc:407] Connection header received
0924 07:12:48.592460 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:48.592461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:48.592471 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:48.592532 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:48.593507 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:48.594015 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:48.595054 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:48.595249 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:48.595313 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:48.597780 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:48.597781 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:48.597782 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:48.597789 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:48.599410 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:48.599929 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:48.599930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:48.599931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:48.600025 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:48.600383 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:48.600384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:48.600384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:48.600512 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:48.600593 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:48.600959 (+ 366us) server_negotiation.cc:299] Negotiation successful
0924 07:12:48.601036 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 07:12:48.601377 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:48.591652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.3:45993)
0924 07:12:48.591813 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:48.591847 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:48.592354 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:48.592586 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:48.592589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:48.593222 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:12:48.593410 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:48.593414 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:48.594103 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:48.594105 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:48.594925 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:48.594928 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:48.595344 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:48.595345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:48.595462 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:48.595934 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:12:48.595942 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:12:48.597650 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:12:48.599530 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:48.599538 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:48.599540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:48.599813 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:48.600105 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:12:48.600107 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:12:48.600107 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:12:48.600264 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:12:48.600582 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:12:48.600588 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:12:48.600800 (+ 212us) client_negotiation.cc:769] Sending connection context
0924 07:12:48.600943 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 07:12:48.601126 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":87,"threads_started":1}
I20260924 07:12:48.601874 16750 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37127
I20260924 07:12:48.602002 16750 heartbeater.cc:464] Registering TS with master...
I20260924 07:12:48.602196 16750 heartbeater.cc:511] Master 127.0.98.62:37127 requested a full tablet report, sending...
I20260924 07:12:48.602612 16287 ts_manager.cc:194] Registered new tserver with Master: 5ad5251214014400b3fb8f34d80c0363 (127.0.98.3:45897)
I20260924 07:12:48.602994 16287 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45993
I20260924 07:12:48.606856 16753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:48.599923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47080 (local address 127.0.98.62:37127)
0924 07:12:48.600054 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:48.600056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:48.600193 (+ 137us) server_negotiation.cc:407] Connection header received
0924 07:12:48.600240 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:48.600241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:48.600252 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:48.600313 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:48.601144 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:48.601935 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:48.602908 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:48.603210 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:48.603280 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:48.604429 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:48.604431 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:48.604431 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:48.604438 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:48.605720 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:48.606096 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:48.606097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:48.606097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:48.606157 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:48.606364 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:48.606365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:48.606365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:48.606469 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:48.606638 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:48.606675 (+ 37us) server_negotiation.cc:299] Negotiation successful
0924 07:12:48.606755 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":57,"threads_started":1}
I20260924 07:12:48.607973 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 07:12:48.611029 16217 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:37127)
I20260924 07:12:48.611088 16753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:48.608414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47086 (local address 127.0.98.62:37127)
0924 07:12:48.608492 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:48.608494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:48.608601 (+ 107us) server_negotiation.cc:407] Connection header received
0924 07:12:48.608686 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:48.608687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:48.608775 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:48.608839 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:12:48.608941 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:48.609401 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:48.610173 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:48.610385 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:48.610458 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:48.610621 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:12:48.610675 (+ 54us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 07:12:48.611010 (+ 335us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47086: token expired
Metrics: {"server-negotiator.queue_time_us":56}
W20260924 07:12:48.611135 16753 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47086: token expired
I20260924 07:12:48.617429 16753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:48.611448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47088 (local address 127.0.98.62:37127)
0924 07:12:48.611485 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:48.611486 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:48.611645 (+ 159us) server_negotiation.cc:407] Connection header received
0924 07:12:48.611707 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:48.611708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:48.611719 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:48.611780 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:12:48.612248 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:48.613016 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:48.613819 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:48.614069 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:48.614125 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:48.615239 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:12:48.615240 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:12:48.615240 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:12:48.615247 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:12:48.616416 (+ 1169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:48.616707 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:48.616707 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:48.616708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:48.616811 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:12:48.616981 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:12:48.616982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:12:48.616982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:12:48.617111 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:12:48.617185 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:48.617279 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 07:12:48.617333 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 07:12:48.618482 16287 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47088:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:12:48.620023 16287 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:12:48.622164 16548 tablet_service.cc:1467] Processing CreateTablet for tablet 58654b12f8ed4f338d97c586f12dfc67 (DEFAULT_TABLE table=test-table [id=af69528dfb0145daa1ff11556f7e8e9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:12:48.622359 16548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58654b12f8ed4f338d97c586f12dfc67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:12:48.622450 16409 tablet_service.cc:1467] Processing CreateTablet for tablet 58654b12f8ed4f338d97c586f12dfc67 (DEFAULT_TABLE table=test-table [id=af69528dfb0145daa1ff11556f7e8e9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:12:48.622623 16409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58654b12f8ed4f338d97c586f12dfc67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:12:48.623633 16506 tablet_bootstrap.cc:492] T 58654b12f8ed4f338d97c586f12dfc67 P 67d8d14af05940f0b8408e3b8162d1a1: Bootstrap starting.
I20260924 07:12:48.623957 16506 tablet_bootstrap.cc:654] T 58654b12f8ed4f338d97c586f12dfc67 P 67d8d14af05940f0b8408e3b8162d1a1: Neither blocks nor log segments found. Creating new log.
I20260924 07:12:48.624100 16367 tablet_bootstrap.cc:492] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d: Bootstrap starting.
I20260924 07:12:48.624369 16367 tablet_bootstrap.cc:654] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d: Neither blocks nor log segments found. Creating new log.
I20260924 07:12:48.624476 16506 tablet_bootstrap.cc:492] T 58654b12f8ed4f338d97c586f12dfc67 P 67d8d14af05940f0b8408e3b8162d1a1: No bootstrap required, opened a new log
I20260924 07:12:48.624521 16506 ts_tablet_manager.cc:1397] T 58654b12f8ed4f338d97c586f12dfc67 P 67d8d14af05940f0b8408e3b8162d1a1: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 07:12:48.624650 16506 raft_consensus.cc:348] T 58654b12f8ed4f338d97c586f12dfc67 P 67d8d14af05940f0b8408e3b8162d1a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad5251214014400b3fb8f34d80c0363" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45897 } } peers { permanent_uuid: "67d8d14af05940f0b8408e3b8162d1a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35475 } } peers { permanent_uuid: "54431bf7bec5412483921836103ad70d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43979 } }
I20260924 07:12:48.624754 16506 raft_consensus.cc:374] T 58654b12f8ed4f338d97c586f12dfc67 P 67d8d14af05940f0b8408e3b8162d1a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:12:48.624804 16506 raft_consensus.cc:729] T 58654b12f8ed4f338d97c586f12dfc67 P 67d8d14af05940f0b8408e3b8162d1a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67d8d14af05940f0b8408e3b8162d1a1, State: Initialized, Role: FOLLOWER
I20260924 07:12:48.624853 16506 consensus_queue.cc:260] T 58654b12f8ed4f338d97c586f12dfc67 P 67d8d14af05940f0b8408e3b8162d1a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad5251214014400b3fb8f34d80c0363" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45897 } } peers { permanent_uuid: "67d8d14af05940f0b8408e3b8162d1a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35475 } } peers { permanent_uuid: "54431bf7bec5412483921836103ad70d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43979 } }
I20260924 07:12:48.624941 16367 tablet_bootstrap.cc:492] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d: No bootstrap required, opened a new log
I20260924 07:12:48.624948 16506 ts_tablet_manager.cc:1428] T 58654b12f8ed4f338d97c586f12dfc67 P 67d8d14af05940f0b8408e3b8162d1a1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:48.624984 16367 ts_tablet_manager.cc:1397] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 07:12:48.625084 16367 raft_consensus.cc:348] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad5251214014400b3fb8f34d80c0363" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45897 } } peers { permanent_uuid: "67d8d14af05940f0b8408e3b8162d1a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35475 } } peers { permanent_uuid: "54431bf7bec5412483921836103ad70d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43979 } }
I20260924 07:12:48.625147 16367 raft_consensus.cc:374] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:12:48.625182 16367 raft_consensus.cc:729] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54431bf7bec5412483921836103ad70d, State: Initialized, Role: FOLLOWER
I20260924 07:12:48.625228 16367 consensus_queue.cc:260] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad5251214014400b3fb8f34d80c0363" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45897 } } peers { permanent_uuid: "67d8d14af05940f0b8408e3b8162d1a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35475 } } peers { permanent_uuid: "54431bf7bec5412483921836103ad70d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43979 } }
I20260924 07:12:48.625301 16367 ts_tablet_manager.cc:1428] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 07:12:48.626096 16482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:48.621935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45897 (local address 127.0.0.1:46114)
0924 07:12:48.621969 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:48.621980 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:48.622015 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:48.622431 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:48.622432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:48.622435 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:12:48.622473 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:48.622474 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:48.623694 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:48.623695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:48.624802 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:48.624806 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:48.625850 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:48.625851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:48.625915 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:48.625919 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:12:48.625991 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 07:12:48.626033 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260924 07:12:48.626142 16756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:48.622060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46114 (local address 127.0.98.3:45897)
0924 07:12:48.622251 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:48.622255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:48.622266 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:12:48.622310 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:48.622313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:48.622329 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:48.622417 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:12:48.622566 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:48.623593 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:48.624898 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:48.625750 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:48.625826 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:48.625887 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:48.626009 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 07:12:48.626049 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":120,"threads_started":1}
I20260924 07:12:48.626880 16684 tablet_service.cc:1467] Processing CreateTablet for tablet 58654b12f8ed4f338d97c586f12dfc67 (DEFAULT_TABLE table=test-table [id=af69528dfb0145daa1ff11556f7e8e9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:12:48.627277 16684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58654b12f8ed4f338d97c586f12dfc67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:12:48.630021 16759 tablet_bootstrap.cc:492] T 58654b12f8ed4f338d97c586f12dfc67 P 5ad5251214014400b3fb8f34d80c0363: Bootstrap starting.
I20260924 07:12:48.630664 16759 tablet_bootstrap.cc:654] T 58654b12f8ed4f338d97c586f12dfc67 P 5ad5251214014400b3fb8f34d80c0363: Neither blocks nor log segments found. Creating new log.
I20260924 07:12:48.631304 16759 log.cc:824] T 58654b12f8ed4f338d97c586f12dfc67 P 5ad5251214014400b3fb8f34d80c0363: Log is configured to *not* fsync() on all Append() calls
I20260924 07:12:48.632122 16759 tablet_bootstrap.cc:492] T 58654b12f8ed4f338d97c586f12dfc67 P 5ad5251214014400b3fb8f34d80c0363: No bootstrap required, opened a new log
I20260924 07:12:48.632189 16759 ts_tablet_manager.cc:1397] T 58654b12f8ed4f338d97c586f12dfc67 P 5ad5251214014400b3fb8f34d80c0363: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:12:48.633569 16759 raft_consensus.cc:348] T 58654b12f8ed4f338d97c586f12dfc67 P 5ad5251214014400b3fb8f34d80c0363 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad5251214014400b3fb8f34d80c0363" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45897 } } peers { permanent_uuid: "67d8d14af05940f0b8408e3b8162d1a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35475 } } peers { permanent_uuid: "54431bf7bec5412483921836103ad70d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43979 } }
I20260924 07:12:48.633701 16759 raft_consensus.cc:374] T 58654b12f8ed4f338d97c586f12dfc67 P 5ad5251214014400b3fb8f34d80c0363 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:12:48.633723 16759 raft_consensus.cc:729] T 58654b12f8ed4f338d97c586f12dfc67 P 5ad5251214014400b3fb8f34d80c0363 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ad5251214014400b3fb8f34d80c0363, State: Initialized, Role: FOLLOWER
I20260924 07:12:48.633833 16759 consensus_queue.cc:260] T 58654b12f8ed4f338d97c586f12dfc67 P 5ad5251214014400b3fb8f34d80c0363 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad5251214014400b3fb8f34d80c0363" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45897 } } peers { permanent_uuid: "67d8d14af05940f0b8408e3b8162d1a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35475 } } peers { permanent_uuid: "54431bf7bec5412483921836103ad70d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43979 } }
I20260924 07:12:48.634112 16759 ts_tablet_manager.cc:1428] T 58654b12f8ed4f338d97c586f12dfc67 P 5ad5251214014400b3fb8f34d80c0363: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 07:12:48.634138 16750 heartbeater.cc:503] Master 127.0.98.62:37127 was elected leader, sending a full tablet report...
I20260924 07:12:48.648794 16481 raft_consensus.cc:482] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:12:48.648952 16481 raft_consensus.cc:504] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad5251214014400b3fb8f34d80c0363" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45897 } } peers { permanent_uuid: "67d8d14af05940f0b8408e3b8162d1a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35475 } } peers { permanent_uuid: "54431bf7bec5412483921836103ad70d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43979 } }
I20260924 07:12:48.649305 16481 leader_election.cc:288] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ad5251214014400b3fb8f34d80c0363 (127.0.98.3:45897), 67d8d14af05940f0b8408e3b8162d1a1 (127.0.98.2:35475)
I20260924 07:12:48.652674 16756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:48.649532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54879 (local address 127.0.98.3:45897)
0924 07:12:48.649573 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:48.649574 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:48.649582 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:12:48.649608 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:48.649608 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:48.649619 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:48.649679 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:12:48.649828 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:48.650514 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:48.651758 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:48.652296 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:48.652347 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:48.652355 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:48.652594 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 07:12:48.652629 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 07:12:48.652678 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:48.649440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45897 (local address 127.0.98.1:54879)
0924 07:12:48.649496 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:48.649508 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:48.649552 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:48.649686 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:48.649687 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:48.649688 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:12:48.649749 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:48.649751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:48.650587 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:48.650588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:48.651684 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:48.651687 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:48.652405 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:48.652406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:48.652476 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:48.652480 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:12:48.652541 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 07:12:48.652584 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 07:12:48.653012 16704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58654b12f8ed4f338d97c586f12dfc67" candidate_uuid: "54431bf7bec5412483921836103ad70d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ad5251214014400b3fb8f34d80c0363" is_pre_election: true
I20260924 07:12:48.653136 16704 raft_consensus.cc:2454] T 58654b12f8ed4f338d97c586f12dfc67 P 5ad5251214014400b3fb8f34d80c0363 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54431bf7bec5412483921836103ad70d in term 0.
I20260924 07:12:48.653367 16361 leader_election.cc:302] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54431bf7bec5412483921836103ad70d, 5ad5251214014400b3fb8f34d80c0363; no voters:
I20260924 07:12:48.653487 16620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:48.649573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42543 (local address 127.0.98.2:35475)
0924 07:12:48.649621 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:48.649623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:48.649730 (+ 107us) server_negotiation.cc:407] Connection header received
0924 07:12:48.649779 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:48.649780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:48.649793 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:48.649866 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:12:48.650316 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:48.650944 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:48.652170 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:48.652667 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:48.652753 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:48.652765 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:48.653374 (+ 609us) server_negotiation.cc:299] Negotiation successful
0924 07:12:48.653423 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 07:12:48.653496 16762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:48.649516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35475 (local address 127.0.98.1:42543)
0924 07:12:48.649662 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:48.649671 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:12:48.649720 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:12:48.649859 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:12:48.649859 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:12:48.649860 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:12:48.650239 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:48.650242 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:48.651029 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:48.651030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:48.652087 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:12:48.652090 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:12:48.652870 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:12:48.652871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:12:48.653299 (+ 428us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:48.653304 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:12:48.653364 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 07:12:48.653397 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
I20260924 07:12:48.653498 16481 raft_consensus.cc:2781] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:12:48.653574 16481 raft_consensus.cc:482] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:12:48.653595 16481 raft_consensus.cc:3037] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:12:48.654268 16481 raft_consensus.cc:504] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad5251214014400b3fb8f34d80c0363" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45897 } } peers { permanent_uuid: "67d8d14af05940f0b8408e3b8162d1a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35475 } } peers { permanent_uuid: "54431bf7bec5412483921836103ad70d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43979 } }
I20260924 07:12:48.654419 16481 leader_election.cc:288] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [CANDIDATE]: Term 1 election: Requested vote from peers 5ad5251214014400b3fb8f34d80c0363 (127.0.98.3:45897), 67d8d14af05940f0b8408e3b8162d1a1 (127.0.98.2:35475)
I20260924 07:12:48.654608 16704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58654b12f8ed4f338d97c586f12dfc67" candidate_uuid: "54431bf7bec5412483921836103ad70d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ad5251214014400b3fb8f34d80c0363"
I20260924 07:12:48.654619 16568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58654b12f8ed4f338d97c586f12dfc67" candidate_uuid: "54431bf7bec5412483921836103ad70d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67d8d14af05940f0b8408e3b8162d1a1" is_pre_election: true
I20260924 07:12:48.654698 16704 raft_consensus.cc:3037] T 58654b12f8ed4f338d97c586f12dfc67 P 5ad5251214014400b3fb8f34d80c0363 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:12:48.654745 16568 raft_consensus.cc:2454] T 58654b12f8ed4f338d97c586f12dfc67 P 67d8d14af05940f0b8408e3b8162d1a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54431bf7bec5412483921836103ad70d in term 0.
I20260924 07:12:48.654831 16567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58654b12f8ed4f338d97c586f12dfc67" candidate_uuid: "54431bf7bec5412483921836103ad70d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67d8d14af05940f0b8408e3b8162d1a1"
I20260924 07:12:48.654924 16567 raft_consensus.cc:3037] T 58654b12f8ed4f338d97c586f12dfc67 P 67d8d14af05940f0b8408e3b8162d1a1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:12:48.655419 16704 raft_consensus.cc:2454] T 58654b12f8ed4f338d97c586f12dfc67 P 5ad5251214014400b3fb8f34d80c0363 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54431bf7bec5412483921836103ad70d in term 1.
I20260924 07:12:48.655580 16361 leader_election.cc:302] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54431bf7bec5412483921836103ad70d, 5ad5251214014400b3fb8f34d80c0363; no voters:
I20260924 07:12:48.655678 16481 raft_consensus.cc:2781] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:12:48.655712 16567 raft_consensus.cc:2454] T 58654b12f8ed4f338d97c586f12dfc67 P 67d8d14af05940f0b8408e3b8162d1a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54431bf7bec5412483921836103ad70d in term 1.
I20260924 07:12:48.655807 16481 raft_consensus.cc:686] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [term 1 LEADER]: Becoming Leader. State: Replica: 54431bf7bec5412483921836103ad70d, State: Running, Role: LEADER
I20260924 07:12:48.655875 16481 consensus_queue.cc:237] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad5251214014400b3fb8f34d80c0363" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45897 } } peers { permanent_uuid: "67d8d14af05940f0b8408e3b8162d1a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35475 } } peers { permanent_uuid: "54431bf7bec5412483921836103ad70d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43979 } }
I20260924 07:12:48.656595 16286 catalog_manager.cc:5495] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d reported cstate change: term changed from 0 to 1, leader changed from <none> to 54431bf7bec5412483921836103ad70d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "54431bf7bec5412483921836103ad70d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad5251214014400b3fb8f34d80c0363" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67d8d14af05940f0b8408e3b8162d1a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54431bf7bec5412483921836103ad70d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43979 } health_report { overall_health: HEALTHY } } }
I20260924 07:12:48.663347 16483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:48.661273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36682 (local address 127.0.98.1:43979)
0924 07:12:48.661344 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 07:12:48.661345 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:12:48.661354 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:12:48.661391 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:12:48.661392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:12:48.661408 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:12:48.661474 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:12:48.661582 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:48.662012 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:48.662693 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:12:48.662866 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:12:48.662933 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:12:48.663104 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:12:48.663190 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:12:48.663244 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 07:12:48.663285 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 07:12:48.663302 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 07:12:48.664776 16704 raft_consensus.cc:1261] T 58654b12f8ed4f338d97c586f12dfc67 P 5ad5251214014400b3fb8f34d80c0363 [term 1 FOLLOWER]: Refusing update from remote peer 54431bf7bec5412483921836103ad70d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:12:48.664806 16567 raft_consensus.cc:1261] T 58654b12f8ed4f338d97c586f12dfc67 P 67d8d14af05940f0b8408e3b8162d1a1 [term 1 FOLLOWER]: Refusing update from remote peer 54431bf7bec5412483921836103ad70d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:12:48.665151 16481 consensus_queue.cc:1035] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ad5251214014400b3fb8f34d80c0363" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:12:48.665277 16763 consensus_queue.cc:1035] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [LEADER]: Connected to new peer: Peer: permanent_uuid: "67d8d14af05940f0b8408e3b8162d1a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:12:48.668107 16766 mvcc.cc:205] Tried to move back new op lower bound from 7332798335648550912 to 7332798335614939136. Current Snapshot: MvccSnapshot[applied={T|T < 7332798335648550912 or (T in {7332798335648550912})}]
I20260924 07:12:48.669471 16286 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:12:48.669557 16286 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:12:48.670274 16286 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc: Sending DeleteTablet for 3 replicas of tablet 58654b12f8ed4f338d97c586f12dfc67
I20260924 07:12:48.670558 16684 tablet_service.cc:1514] Processing DeleteTablet for tablet 58654b12f8ed4f338d97c586f12dfc67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:12:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46114
I20260924 07:12:48.670647 16548 tablet_service.cc:1514] Processing DeleteTablet for tablet 58654b12f8ed4f338d97c586f12dfc67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:12:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60562
I20260924 07:12:48.670792 16621 tablet_replica.cc:330] T 58654b12f8ed4f338d97c586f12dfc67 P 67d8d14af05940f0b8408e3b8162d1a1: stopping tablet replica
I20260924 07:12:48.670786 16409 tablet_service.cc:1514] Processing DeleteTablet for tablet 58654b12f8ed4f338d97c586f12dfc67 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:12:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36672
I20260924 07:12:48.670823 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 16325
I20260924 07:12:48.670866 16484 tablet_replica.cc:330] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d: stopping tablet replica
I20260924 07:12:48.670852 16621 raft_consensus.cc:2229] T 58654b12f8ed4f338d97c586f12dfc67 P 67d8d14af05940f0b8408e3b8162d1a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:12:48.670907 16484 raft_consensus.cc:2229] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:12:48.670918 16621 raft_consensus.cc:2258] T 58654b12f8ed4f338d97c586f12dfc67 P 67d8d14af05940f0b8408e3b8162d1a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:12:48.670977 16484 raft_consensus.cc:2258] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:12:48.671211 16621 ts_tablet_manager.cc:1905] T 58654b12f8ed4f338d97c586f12dfc67 P 67d8d14af05940f0b8408e3b8162d1a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:12:48.673041 16621 ts_tablet_manager.cc:1918] T 58654b12f8ed4f338d97c586f12dfc67 P 67d8d14af05940f0b8408e3b8162d1a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:12:48.673061 16776 tablet_replica.cc:330] T 58654b12f8ed4f338d97c586f12dfc67 P 5ad5251214014400b3fb8f34d80c0363: stopping tablet replica
I20260924 07:12:48.673094 16621 log.cc:1196] T 58654b12f8ed4f338d97c586f12dfc67 P 67d8d14af05940f0b8408e3b8162d1a1: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/wal/wals/58654b12f8ed4f338d97c586f12dfc67
I20260924 07:12:48.673136 16776 raft_consensus.cc:2229] T 58654b12f8ed4f338d97c586f12dfc67 P 5ad5251214014400b3fb8f34d80c0363 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:12:48.673218 16776 raft_consensus.cc:2258] T 58654b12f8ed4f338d97c586f12dfc67 P 5ad5251214014400b3fb8f34d80c0363 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:12:48.673313 16621 ts_tablet_manager.cc:1939] T 58654b12f8ed4f338d97c586f12dfc67 P 67d8d14af05940f0b8408e3b8162d1a1: Deleting consensus metadata
I20260924 07:12:48.673614 16776 ts_tablet_manager.cc:1905] T 58654b12f8ed4f338d97c586f12dfc67 P 5ad5251214014400b3fb8f34d80c0363: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:12:48.673851 16273 catalog_manager.cc:4841] TS 67d8d14af05940f0b8408e3b8162d1a1 (127.0.98.2:35475): tablet 58654b12f8ed4f338d97c586f12dfc67 (table test-table [id=af69528dfb0145daa1ff11556f7e8e9b]) successfully deleted
I20260924 07:12:48.674947 16776 ts_tablet_manager.cc:1918] T 58654b12f8ed4f338d97c586f12dfc67 P 5ad5251214014400b3fb8f34d80c0363: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:12:48.675034 16776 log.cc:1196] T 58654b12f8ed4f338d97c586f12dfc67 P 5ad5251214014400b3fb8f34d80c0363: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/wal/wals/58654b12f8ed4f338d97c586f12dfc67
I20260924 07:12:48.675312 16776 ts_tablet_manager.cc:1939] T 58654b12f8ed4f338d97c586f12dfc67 P 5ad5251214014400b3fb8f34d80c0363: Deleting consensus metadata
I20260924 07:12:48.675738 16271 catalog_manager.cc:4841] TS 5ad5251214014400b3fb8f34d80c0363 (127.0.98.3:45897): tablet 58654b12f8ed4f338d97c586f12dfc67 (table test-table [id=af69528dfb0145daa1ff11556f7e8e9b]) successfully deleted
W20260924 07:12:48.677335 16271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43979 (error 108)
I20260924 07:12:48.677649 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 16480
I20260924 07:12:48.677852 16482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:12:48.677728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43979 (local address 127.0.0.1:36696)
0924 07:12:48.677777 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 07:12:48.677814 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 07:12:48.679168 16271 catalog_manager.cc:4584] TS 54431bf7bec5412483921836103ad70d (127.0.98.1:43979): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 58654b12f8ed4f338d97c586f12dfc67: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43979: connect: Connection refused (error 111)
I20260924 07:12:48.682394 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 16617
I20260924 07:12:48.686990 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 16254
I20260924 07:13:08.692514 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37127
--webserver_interface=127.0.98.62
--webserver_port=41239
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:08.771603 16778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:08.771790 16778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:08.771812 16778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:08.772966 16778 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 07:13:08.773005 16778 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 07:13:08.773017 16778 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 07:13:08.773029 16778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:13:08.773077 16778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:08.773096 16778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:08.773119 16778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:13:08.773147 16778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:13:08.774591 16778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:08.774809 16778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:08.775033 16778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:08.780130 16783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:08.780174 16784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:08.780190 16778 server_base.cc:1034] running on GCE node
W20260924 07:13:08.780288 16786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:08.780568 16778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:08.780880 16778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:08.782021 16778 hybrid_clock.cc:648] HybridClock initialized: now 1790233988782002 us; error 39 us; skew 500 ppm
Sep 24 07:13:08 dist-test-slave-14ww krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:08.784612 16778 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:13:08.785063 16778 webserver.cc:460] Webserver started at http://127.0.98.62:41239/ using document root <none> and password file <none>
I20260924 07:13:08.785296 16778 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:08.785341 16778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:08.786612 16778 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:13:08.787325 16793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:08.787515 16778 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:13:08.787586 16778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "fc6625c0c7d047e594a80ff678e26bdc"
format_stamp: "Formatted at 2026-09-24 07:12:27 on dist-test-slave-14ww"
server_key: "782ee73060c760169f463f8057bac125"
server_key_iv: "086b8c1a314427f9c102325a87a593d4"
server_key_version: "encryptionkey@0"
I20260924 07:13:08.787905 16778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:08.799276 16778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:08.801302 16778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:08.801445 16778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:08.805553 16778 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37127
I20260924 07:13:08.805599 16845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37127 every 8 connection(s)
I20260924 07:13:08.805964 16778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:13:08.807080 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 16778
I20260924 07:13:08.807390 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43979
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37127
--webserver_port=36427
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:13:08.807673 16846 sys_catalog.cc:263] Verifying existing consensus state
I20260924 07:13:08.808785 16846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc: Bootstrap starting.
I20260924 07:13:08.817117 16846 log.cc:824] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:08.819336 16846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:13:08.819651 16846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc: Bootstrap complete.
I20260924 07:13:08.822124 16846 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6625c0c7d047e594a80ff678e26bdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37127 } }
I20260924 07:13:08.822345 16846 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc6625c0c7d047e594a80ff678e26bdc, State: Initialized, Role: FOLLOWER
I20260924 07:13:08.822510 16846 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6625c0c7d047e594a80ff678e26bdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37127 } }
I20260924 07:13:08.822566 16846 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:13:08.822628 16846 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:13:08.822679 16846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [term 2 FOLLOWER]: Advancing to term 3
I20260924 07:13:08.823632 16846 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6625c0c7d047e594a80ff678e26bdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37127 } }
I20260924 07:13:08.823714 16846 leader_election.cc:302] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc6625c0c7d047e594a80ff678e26bdc; no voters:
I20260924 07:13:08.823939 16846 leader_election.cc:288] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 07:13:08.823973 16850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [term 3 FOLLOWER]: Leader election won for term 3
I20260924 07:13:08.824085 16850 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [term 3 LEADER]: Becoming Leader. State: Replica: fc6625c0c7d047e594a80ff678e26bdc, State: Running, Role: LEADER
I20260924 07:13:08.824208 16846 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:13:08.824226 16850 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6625c0c7d047e594a80ff678e26bdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37127 } }
I20260924 07:13:08.824621 16852 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc6625c0c7d047e594a80ff678e26bdc. Latest consensus state: current_term: 3 leader_uuid: "fc6625c0c7d047e594a80ff678e26bdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6625c0c7d047e594a80ff678e26bdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37127 } } }
I20260924 07:13:08.824760 16852 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:08.824978 16851 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "fc6625c0c7d047e594a80ff678e26bdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc6625c0c7d047e594a80ff678e26bdc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37127 } } }
I20260924 07:13:08.825067 16851 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:08.825167 16856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:13:08.825941 16856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:13:08.826041 16856 catalog_manager.cc:1259] Loaded cluster ID: 206e4d8a723e4869b8c0297eab5c66d0
I20260924 07:13:08.826122 16856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:13:08.827515 16856 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:13:08.827728 16856 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc: Loaded TSK: 0
I20260924 07:13:08.827807 16856 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fc6625c0c7d047e594a80ff678e26bdc: Loaded TSK: 1
F20260924 07:13:08.828074 16856 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9c950b79ad google::LogMessage::Fail() at ??:0
@ 0x7f9c950ba88a google::LogMessage::SendToLog() at ??:0
@ 0x7f9c950b74de google::LogMessage::Flush() at ??:0
@ 0x7f9c950b8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9c97824104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f9c97821d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f9c9d0111cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f9c9d01148e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f9c9d011594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f9c9cfd7c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f9c9cfe1ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9c96f850d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9c96f7a954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9c97dde6db start_thread at ??:0
@ 0x7f9c93bb561f clone at ??:0
W20260924 07:13:08.906407 16848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:08.906607 16848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:08.906631 16848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:08.907814 16848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:08.907855 16848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:08.907882 16848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:13:08.909405 16848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43979
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36427
--tserver_master_addrs=127.0.98.62:37127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:08.909644 16848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:08.909902 16848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:08.916083 16887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:08.916117 16888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:08.916079 16890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:08.916514 16848 server_base.cc:1034] running on GCE node
I20260924 07:13:08.916715 16848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:08.917014 16848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:08.918174 16848 hybrid_clock.cc:648] HybridClock initialized: now 1790233988918156 us; error 37 us; skew 500 ppm
Sep 24 07:13:08 dist-test-slave-14ww krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:08.920878 16848 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:13:08.921363 16848 webserver.cc:460] Webserver started at http://127.0.98.1:36427/ using document root <none> and password file <none>
I20260924 07:13:08.921599 16848 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:08.921669 16848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:08.923055 16848 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:13:08.923745 16897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:08.923892 16848 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:13:08.923995 16848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/wal
uuid: "54431bf7bec5412483921836103ad70d"
format_stamp: "Formatted at 2026-09-24 07:12:27 on dist-test-slave-14ww"
server_key: "49aeb66134fa49a1842c4c882ef8d5a7"
server_key_iv: "f4d1f6c520ee3b1b71fc447996c32afd"
server_key_version: "encryptionkey@0"
I20260924 07:13:08.924341 16848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:08.951162 16848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:08.953689 16848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:08.953867 16848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:08.954149 16848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:13:08.954734 16904 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 07:13:08.956395 16848 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 07:13:08.956445 16848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 07:13:08.956481 16848 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 07:13:08.957234 16848 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 07:13:08.957280 16848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 07:13:08.957327 16904 tablet_bootstrap.cc:492] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d: Bootstrap starting.
I20260924 07:13:08.966025 16848 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43979
I20260924 07:13:08.967051 16848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:13:08.967988 17011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43979 every 8 connection(s)
I20260924 07:13:08.969441 16904 log.cc:824] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:08.971637 16904 tablet_bootstrap.cc:492] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 07:13:08.971922 16904 tablet_bootstrap.cc:492] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d: Bootstrap complete.
I20260924 07:13:08.972798 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 16848
I20260924 07:13:08.972946 16904 ts_tablet_manager.cc:1397] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.015s
I20260924 07:13:08.973104 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35475
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37127
--webserver_port=45559
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:13:08.974056 16904 raft_consensus.cc:348] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad5251214014400b3fb8f34d80c0363" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45897 } } peers { permanent_uuid: "67d8d14af05940f0b8408e3b8162d1a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35475 } } peers { permanent_uuid: "54431bf7bec5412483921836103ad70d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43979 } }
I20260924 07:13:08.974303 16904 raft_consensus.cc:729] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54431bf7bec5412483921836103ad70d, State: Initialized, Role: FOLLOWER
I20260924 07:13:08.974531 16904 consensus_queue.cc:260] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad5251214014400b3fb8f34d80c0363" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45897 } } peers { permanent_uuid: "67d8d14af05940f0b8408e3b8162d1a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35475 } } peers { permanent_uuid: "54431bf7bec5412483921836103ad70d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43979 } }
I20260924 07:13:08.974828 16904 ts_tablet_manager.cc:1428] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260924 07:13:09.021970 17014 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:13:08.967853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.1:44251)
0924 07:13:08.968071 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:08.968131 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:08.968813 (+ 682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:09.021740 (+ 52927us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":113,"threads_started":1}
W20260924 07:13:09.022271 17012 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37127 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37127: Client connection negotiation failed: client connection to 127.0.98.62:37127: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 07:13:09.022636 17014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:09.022520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.1:35553)
0924 07:13:09.022573 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:09.022608 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 07:13:09.022992 17014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:09.022910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.1:59759)
0924 07:13:09.022956 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:09.022978 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 07:13:09.023113 17012 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 07:13:09.055713 17016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:09.055936 17016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:09.055975 17016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:09.057156 17016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:09.057216 17016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:09.057261 17016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:13:09.058703 17016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35475
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=45559
--tserver_master_addrs=127.0.98.62:37127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:09.058955 17016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:09.059172 17016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:09.064188 17022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:09.064250 17025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:09.064257 17023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:09.064427 17016 server_base.cc:1034] running on GCE node
I20260924 07:13:09.064625 17016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:09.064884 17016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:09.066045 17016 hybrid_clock.cc:648] HybridClock initialized: now 1790233989066026 us; error 33 us; skew 500 ppm
Sep 24 07:13:09 dist-test-slave-14ww krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:09.068578 17016 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:13:09.069042 17016 webserver.cc:460] Webserver started at http://127.0.98.2:45559/ using document root <none> and password file <none>
I20260924 07:13:09.069262 17016 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:09.069337 17016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:09.070667 17016 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:13:09.071221 17032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:09.071388 17016 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:09.071506 17016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/wal
uuid: "67d8d14af05940f0b8408e3b8162d1a1"
format_stamp: "Formatted at 2026-09-24 07:12:27 on dist-test-slave-14ww"
server_key: "d33af9cf35d87548fe4aa025c70e1d79"
server_key_iv: "5c5a48d2cdde09b55f42b9ef12d62fb3"
server_key_version: "encryptionkey@0"
I20260924 07:13:09.071828 17016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:09.092942 17016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:09.094748 17016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:09.094914 17016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:09.095146 17016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:13:09.095448 17016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:13:09.095503 17016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:09.095544 17016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:13:09.095574 17016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:09.101368 17016 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35475
I20260924 07:13:09.101418 17145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35475 every 8 connection(s)
I20260924 07:13:09.101732 17016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:13:09.102685 17148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:09.102339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.2:55383)
0924 07:13:09.102489 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:09.102552 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
W20260924 07:13:09.103056 17146 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37127 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37127: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
I20260924 07:13:09.103299 17148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:09.103221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.2:51641)
0924 07:13:09.103247 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:09.103283 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
I20260924 07:13:09.103543 17148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:09.103494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.2:58281)
0924 07:13:09.103515 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:09.103532 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260924 07:13:09.103649 17146 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 07:13:09.108834 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 17016
I20260924 07:13:09.109107 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45897
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37127
--webserver_port=33445
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:09.189679 17149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:09.189882 17149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:09.189905 17149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:09.190996 17149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:09.191032 17149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:09.191057 17149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:13:09.192409 17149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45897
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33445
--tserver_master_addrs=127.0.98.62:37127
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:09.192589 17149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:09.192782 17149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:09.197815 17154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:09.197885 17155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:09.197980 17149 server_base.cc:1034] running on GCE node
W20260924 07:13:09.197886 17157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:09.198223 17149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:09.198463 17149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:09.199635 17149 hybrid_clock.cc:648] HybridClock initialized: now 1790233989199607 us; error 39 us; skew 500 ppm
Sep 24 07:13:09 dist-test-slave-14ww krb5kdc[15710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:09.202212 17149 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:13:09.202620 17149 webserver.cc:460] Webserver started at http://127.0.98.3:33445/ using document root <none> and password file <none>
I20260924 07:13:09.202841 17149 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:09.202910 17149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:09.204237 17149 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:13:09.204946 17164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:09.205132 17149 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:13:09.205228 17149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/wal
uuid: "5ad5251214014400b3fb8f34d80c0363"
format_stamp: "Formatted at 2026-09-24 07:12:27 on dist-test-slave-14ww"
server_key: "6d835bf92972584cf86fe74a0ff83cad"
server_key_iv: "9cbcd799cca54ea60b5af819b191b64a"
server_key_version: "encryptionkey@0"
I20260924 07:13:09.205564 17149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:09.221010 17149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:09.222806 17149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:09.222967 17149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:09.223189 17149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:13:09.223488 17149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:13:09.223542 17149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:09.223574 17149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:13:09.223603 17149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:09.229516 17149 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45897
I20260924 07:13:09.229550 17277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45897 every 8 connection(s)
I20260924 07:13:09.229897 17149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:13:09.230890 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:09.230549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.3:55289)
0924 07:13:09.230689 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:09.230755 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
W20260924 07:13:09.231282 17278 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37127 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37127: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
I20260924 07:13:09.231549 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:09.231464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.3:56953)
0924 07:13:09.231508 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:09.231533 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 07:13:09.231863 17280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:09.231787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.3:51057)
0924 07:13:09.231828 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:09.231850 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 07:13:09.231976 17278 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 07:13:09.234237 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 17149
I20260924 07:13:10.023663 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:10.023446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.1:39583)
0924 07:13:10.023583 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:10.023616 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
I20260924 07:13:10.104101 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:10.103928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.2:36203)
0924 07:13:10.104030 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:10.104060 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":54,"threads_started":1}
I20260924 07:13:10.232515 17284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:10.232302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.3:47543)
0924 07:13:10.232429 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:10.232463 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20260924 07:13:10.318151 17285 raft_consensus.cc:482] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:13:10.318275 17285 raft_consensus.cc:504] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad5251214014400b3fb8f34d80c0363" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45897 } } peers { permanent_uuid: "67d8d14af05940f0b8408e3b8162d1a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35475 } } peers { permanent_uuid: "54431bf7bec5412483921836103ad70d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43979 } }
I20260924 07:13:10.318681 17285 leader_election.cc:288] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ad5251214014400b3fb8f34d80c0363 (127.0.98.3:45897), 67d8d14af05940f0b8408e3b8162d1a1 (127.0.98.2:35475)
Sep 24 07:13:10 dist-test-slave-14ww krb5kdc[15710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 24 07:13:10 dist-test-slave-14ww krb5kdc[15710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790233988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260924 07:13:10.329680 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:10.318836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45897 (local address 127.0.98.1:48463)
0924 07:13:10.318895 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:10.318908 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:10.318967 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:10.319874 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:10.319877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:10.320227 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:10.320422 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:10.320428 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:10.321715 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:10.321716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:10.322467 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:10.322469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:10.323032 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:10.323033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:10.323140 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:10.323776 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:10.323781 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:10.325326 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:10.327642 (+ 2316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:10.327647 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:10.327649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:10.328055 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:10.328376 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:10.328377 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:10.328377 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:10.328551 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:10.329073 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:10.329080 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:10.329235 (+ 155us) client_negotiation.cc:769] Sending connection context
0924 07:13:10.329369 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 07:13:10.329564 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":101}
I20260924 07:13:10.329689 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:10.319040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37729 (local address 127.0.98.2:35475)
0924 07:13:10.319183 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:10.319190 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:10.319214 (+ 24us) server_negotiation.cc:407] Connection header received
0924 07:13:10.319633 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:10.319635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:10.319672 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:10.319810 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:10.320537 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:10.321609 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:10.322555 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:10.322940 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:10.323059 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:10.325840 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:10.325842 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:10.325843 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:10.325851 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:10.327651 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:10.328170 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:10.328173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:10.328173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:10.328293 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:10.328647 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:10.328648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:10.328649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:10.328989 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:10.329085 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:10.329442 (+ 357us) server_negotiation.cc:299] Negotiation successful
0924 07:13:10.329574 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":61,"threads_started":1}
I20260924 07:13:10.329692 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:10.318944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48463 (local address 127.0.98.3:45897)
0924 07:13:10.319107 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:10.319118 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:10.319168 (+ 50us) server_negotiation.cc:407] Connection header received
0924 07:13:10.319622 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:10.319624 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:10.319672 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:10.319829 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:10.320539 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:10.321608 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:10.322557 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:10.322939 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:10.323099 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:10.325425 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:10.325428 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:10.325428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:10.325437 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:10.327566 (+ 2129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:10.328156 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:10.328160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:10.328160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:10.328293 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:10.328647 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:10.328648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:10.328649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:10.328989 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:10.329085 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:10.329442 (+ 357us) server_negotiation.cc:299] Negotiation successful
0924 07:13:10.329574 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":55,"threads_started":1}
I20260924 07:13:10.329734 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:10.318888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35475 (local address 127.0.98.1:37729)
0924 07:13:10.319007 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:10.319017 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:10.319089 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:10.319873 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:10.319876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:10.320235 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:10.320422 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:10.320428 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:10.321705 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:10.321706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:10.322466 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:10.322468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:10.323031 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:10.323032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:10.323139 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:10.323710 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:10.323718 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:10.325721 (+ 2003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:10.327763 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:10.327764 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:10.327765 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:10.328067 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:10.328376 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:10.328377 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:10.328377 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:10.328551 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:10.329073 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:10.329080 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:10.329235 (+ 155us) client_negotiation.cc:769] Sending connection context
0924 07:13:10.329379 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 07:13:10.329620 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":112,"thread_start_us":64,"threads_started":1}
W20260924 07:13:10.331302 16900 leader_election.cc:341] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 67d8d14af05940f0b8408e3b8162d1a1 (127.0.98.2:35475): Not found: Tablet not found: 58654b12f8ed4f338d97c586f12dfc67
W20260924 07:13:10.331405 16898 leader_election.cc:341] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ad5251214014400b3fb8f34d80c0363 (127.0.98.3:45897): Not found: Tablet not found: 58654b12f8ed4f338d97c586f12dfc67
I20260924 07:13:10.331511 16898 leader_election.cc:302] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 54431bf7bec5412483921836103ad70d; no voters: 5ad5251214014400b3fb8f34d80c0363, 67d8d14af05940f0b8408e3b8162d1a1
I20260924 07:13:10.331643 17285 raft_consensus.cc:2726] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:13:11.024277 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:11.024077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.1:44883)
0924 07:13:11.024194 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:11.024229 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":65,"threads_started":1}
I20260924 07:13:11.104676 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:11.104488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.2:41713)
0924 07:13:11.104602 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:11.104634 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
I20260924 07:13:11.233203 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:11.233001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.3:53331)
0924 07:13:11.233127 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:11.233162 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":60,"threads_started":1}
I20260924 07:13:12.025213 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:12.024890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.1:56177)
0924 07:13:12.025128 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:12.025166 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":83,"threads_started":1}
I20260924 07:13:12.105340 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:12.105154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.2:33245)
0924 07:13:12.105268 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:12.105302 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":57,"threads_started":1}
I20260924 07:13:12.233827 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:12.233639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.3:59545)
0924 07:13:12.233757 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:12.233788 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":58,"threads_started":1}
I20260924 07:13:12.476914 17295 raft_consensus.cc:482] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:13:12.476996 17295 raft_consensus.cc:504] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad5251214014400b3fb8f34d80c0363" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45897 } } peers { permanent_uuid: "67d8d14af05940f0b8408e3b8162d1a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35475 } } peers { permanent_uuid: "54431bf7bec5412483921836103ad70d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43979 } }
I20260924 07:13:12.477140 17295 leader_election.cc:288] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ad5251214014400b3fb8f34d80c0363 (127.0.98.3:45897), 67d8d14af05940f0b8408e3b8162d1a1 (127.0.98.2:35475)
W20260924 07:13:12.477569 16900 leader_election.cc:341] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 67d8d14af05940f0b8408e3b8162d1a1 (127.0.98.2:35475): Not found: Tablet not found: 58654b12f8ed4f338d97c586f12dfc67
W20260924 07:13:12.477640 16898 leader_election.cc:341] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ad5251214014400b3fb8f34d80c0363 (127.0.98.3:45897): Not found: Tablet not found: 58654b12f8ed4f338d97c586f12dfc67
I20260924 07:13:12.477684 16898 leader_election.cc:302] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 54431bf7bec5412483921836103ad70d; no voters: 5ad5251214014400b3fb8f34d80c0363, 67d8d14af05940f0b8408e3b8162d1a1
I20260924 07:13:12.477767 17295 raft_consensus.cc:2726] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:13:13.026008 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:13.025770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.1:41037)
0924 07:13:13.025926 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:13.025961 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":79,"threads_started":1}
I20260924 07:13:13.106132 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:13.105924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.2:48349)
0924 07:13:13.106056 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:13.106092 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":60,"threads_started":1}
I20260924 07:13:13.234479 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:13.234277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.3:46405)
0924 07:13:13.234412 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:13.234443 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":76,"threads_started":1}
I20260924 07:13:14.026876 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:14.026622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.1:47853)
0924 07:13:14.026781 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:14.026823 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":90,"threads_started":1}
I20260924 07:13:14.106865 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:14.106629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.2:59031)
0924 07:13:14.106776 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:14.106821 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":63,"threads_started":1}
I20260924 07:13:14.235291 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:14.235068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.3:37601)
0924 07:13:14.235215 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:14.235255 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":58,"threads_started":1}
I20260924 07:13:15.027560 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:15.027347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.1:52665)
0924 07:13:15.027481 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:15.027514 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":74,"threads_started":1}
I20260924 07:13:15.107568 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:15.107371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.2:43879)
0924 07:13:15.107495 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:15.107530 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
I20260924 07:13:15.235942 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:15.235732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.3:53075)
0924 07:13:15.235862 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:15.235897 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
I20260924 07:13:15.496716 17305 raft_consensus.cc:482] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:13:15.496861 17305 raft_consensus.cc:504] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad5251214014400b3fb8f34d80c0363" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45897 } } peers { permanent_uuid: "67d8d14af05940f0b8408e3b8162d1a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35475 } } peers { permanent_uuid: "54431bf7bec5412483921836103ad70d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43979 } }
I20260924 07:13:15.497021 17305 leader_election.cc:288] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ad5251214014400b3fb8f34d80c0363 (127.0.98.3:45897), 67d8d14af05940f0b8408e3b8162d1a1 (127.0.98.2:35475)
W20260924 07:13:15.497434 16898 leader_election.cc:341] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ad5251214014400b3fb8f34d80c0363 (127.0.98.3:45897): Not found: Tablet not found: 58654b12f8ed4f338d97c586f12dfc67
W20260924 07:13:15.497520 16900 leader_election.cc:341] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 67d8d14af05940f0b8408e3b8162d1a1 (127.0.98.2:35475): Not found: Tablet not found: 58654b12f8ed4f338d97c586f12dfc67
I20260924 07:13:15.497613 16900 leader_election.cc:302] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 54431bf7bec5412483921836103ad70d; no voters: 5ad5251214014400b3fb8f34d80c0363, 67d8d14af05940f0b8408e3b8162d1a1
I20260924 07:13:15.497699 17305 raft_consensus.cc:2726] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:13:16.028170 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:16.027964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.1:56969)
0924 07:13:16.028079 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:16.028108 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
I20260924 07:13:16.108233 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:16.108045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.2:56405)
0924 07:13:16.108157 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:16.108190 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
I20260924 07:13:16.236583 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:16.236360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.3:51681)
0924 07:13:16.236510 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:16.236546 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":73,"threads_started":1}
I20260924 07:13:17.028959 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:17.028676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.1:49497)
0924 07:13:17.028866 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:17.028906 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":130,"threads_started":1}
I20260924 07:13:17.108963 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:17.108719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.2:58863)
0924 07:13:17.108889 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:17.108923 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":113,"threads_started":1}
I20260924 07:13:17.237277 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:17.237086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.3:60769)
0924 07:13:17.237199 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:17.237235 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260924 07:13:17.326800 17312 raft_consensus.cc:482] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:13:17.326892 17312 raft_consensus.cc:504] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad5251214014400b3fb8f34d80c0363" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45897 } } peers { permanent_uuid: "67d8d14af05940f0b8408e3b8162d1a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35475 } } peers { permanent_uuid: "54431bf7bec5412483921836103ad70d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43979 } }
I20260924 07:13:17.327100 17312 leader_election.cc:288] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ad5251214014400b3fb8f34d80c0363 (127.0.98.3:45897), 67d8d14af05940f0b8408e3b8162d1a1 (127.0.98.2:35475)
W20260924 07:13:17.327478 16900 leader_election.cc:341] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 67d8d14af05940f0b8408e3b8162d1a1 (127.0.98.2:35475): Not found: Tablet not found: 58654b12f8ed4f338d97c586f12dfc67
W20260924 07:13:17.327541 16898 leader_election.cc:341] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ad5251214014400b3fb8f34d80c0363 (127.0.98.3:45897): Not found: Tablet not found: 58654b12f8ed4f338d97c586f12dfc67
I20260924 07:13:17.327569 16900 leader_election.cc:302] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 54431bf7bec5412483921836103ad70d; no voters: 5ad5251214014400b3fb8f34d80c0363, 67d8d14af05940f0b8408e3b8162d1a1
I20260924 07:13:17.327637 17312 raft_consensus.cc:2726] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:13:18.029783 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:18.029514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.1:51105)
0924 07:13:18.029679 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:18.029732 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":67,"threads_started":1}
I20260924 07:13:18.109676 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:18.109473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.2:58981)
0924 07:13:18.109593 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:18.109636 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":56,"threads_started":1}
I20260924 07:13:18.238058 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:18.237787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.3:57669)
0924 07:13:18.237963 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:18.238011 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":83,"threads_started":1}
I20260924 07:13:18.986259 17316 raft_consensus.cc:482] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:13:18.986366 17316 raft_consensus.cc:504] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad5251214014400b3fb8f34d80c0363" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45897 } } peers { permanent_uuid: "67d8d14af05940f0b8408e3b8162d1a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35475 } } peers { permanent_uuid: "54431bf7bec5412483921836103ad70d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43979 } }
I20260924 07:13:18.986539 17316 leader_election.cc:288] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ad5251214014400b3fb8f34d80c0363 (127.0.98.3:45897), 67d8d14af05940f0b8408e3b8162d1a1 (127.0.98.2:35475)
W20260924 07:13:18.986977 16900 leader_election.cc:341] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 67d8d14af05940f0b8408e3b8162d1a1 (127.0.98.2:35475): Not found: Tablet not found: 58654b12f8ed4f338d97c586f12dfc67
W20260924 07:13:18.987057 16898 leader_election.cc:341] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ad5251214014400b3fb8f34d80c0363 (127.0.98.3:45897): Not found: Tablet not found: 58654b12f8ed4f338d97c586f12dfc67
I20260924 07:13:18.987092 16898 leader_election.cc:302] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 54431bf7bec5412483921836103ad70d; no voters: 5ad5251214014400b3fb8f34d80c0363, 67d8d14af05940f0b8408e3b8162d1a1
I20260924 07:13:18.987175 17316 raft_consensus.cc:2726] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:13:19.030592 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:19.030404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.1:56849)
0924 07:13:19.030519 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:19.030549 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
I20260924 07:13:19.110332 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:19.110110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.2:48691)
0924 07:13:19.110239 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:19.110290 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
I20260924 07:13:19.238710 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:19.238516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.3:53583)
0924 07:13:19.238638 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:19.238672 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20260924 07:13:20.031241 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:20.031029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.1:45463)
0924 07:13:20.031158 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:20.031195 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
I20260924 07:13:20.110975 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:20.110767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.2:49497)
0924 07:13:20.110897 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:20.110932 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":58,"threads_started":1}
I20260924 07:13:20.239372 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:20.239181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.3:42057)
0924 07:13:20.239295 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:20.239330 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260924 07:13:21.032001 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:21.031706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.1:54471)
0924 07:13:21.031915 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:21.031953 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":57,"threads_started":1}
I20260924 07:13:21.111603 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:21.111388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.2:58379)
0924 07:13:21.111525 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:21.111559 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":67,"threads_started":1}
I20260924 07:13:21.240126 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:21.239912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.3:60993)
0924 07:13:21.240054 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:21.240087 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":71,"threads_started":1}
I20260924 07:13:22.032720 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:22.032503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.1:46151)
0924 07:13:22.032633 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:22.032671 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":64,"threads_started":1}
I20260924 07:13:22.112179 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:22.111992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.2:60995)
0924 07:13:22.112107 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:22.112139 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":57,"threads_started":1}
I20260924 07:13:22.240865 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:22.240556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.3:59063)
0924 07:13:22.240698 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:22.240767 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":71,"threads_started":1}
I20260924 07:13:23.033403 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:23.033198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.1:43967)
0924 07:13:23.033317 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:23.033356 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":62,"threads_started":1}
I20260924 07:13:23.112926 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:23.112623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.2:42693)
0924 07:13:23.112839 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:23.112875 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":60,"threads_started":1}
I20260924 07:13:23.241529 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:23.241308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.3:55161)
0924 07:13:23.241450 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:23.241488 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":72,"threads_started":1}
I20260924 07:13:24.034140 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:24.033932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.1:50205)
0924 07:13:24.034049 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:24.034081 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
I20260924 07:13:24.113562 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:24.113375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.2:36347)
0924 07:13:24.113498 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:24.113524 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":76,"threads_started":1}
I20260924 07:13:24.151219 17334 raft_consensus.cc:482] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:13:24.151348 17334 raft_consensus.cc:504] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ad5251214014400b3fb8f34d80c0363" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45897 } } peers { permanent_uuid: "67d8d14af05940f0b8408e3b8162d1a1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35475 } } peers { permanent_uuid: "54431bf7bec5412483921836103ad70d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43979 } }
I20260924 07:13:24.151568 17334 leader_election.cc:288] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5ad5251214014400b3fb8f34d80c0363 (127.0.98.3:45897), 67d8d14af05940f0b8408e3b8162d1a1 (127.0.98.2:35475)
W20260924 07:13:24.152048 16898 leader_election.cc:341] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5ad5251214014400b3fb8f34d80c0363 (127.0.98.3:45897): Not found: Tablet not found: 58654b12f8ed4f338d97c586f12dfc67
W20260924 07:13:24.152143 16900 leader_election.cc:341] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 67d8d14af05940f0b8408e3b8162d1a1 (127.0.98.2:35475): Not found: Tablet not found: 58654b12f8ed4f338d97c586f12dfc67
I20260924 07:13:24.152237 16900 leader_election.cc:302] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 54431bf7bec5412483921836103ad70d; no voters: 5ad5251214014400b3fb8f34d80c0363, 67d8d14af05940f0b8408e3b8162d1a1
I20260924 07:13:24.152335 17334 raft_consensus.cc:2726] T 58654b12f8ed4f338d97c586f12dfc67 P 54431bf7bec5412483921836103ad70d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 07:13:24.238250 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 16848
I20260924 07:13:24.242221 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:24.242030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37127 (local address 127.0.98.3:38385)
0924 07:13:24.242146 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:24.242174 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":70,"threads_started":1}
I20260924 07:13:24.243013 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 17016
I20260924 07:13:24.247107 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 17149
2026-09-24T07:13:24Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58376 ms)
[----------] 30 tests from SecurityITest (231846 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:13:24 dist-test-slave-14ww krb5kdc[17338](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:13:24 dist-test-slave-14ww krb5kdc[17338](info): set up 2 sockets
Sep 24 07:13:24 dist-test-slave-14ww krb5kdc[17338](info): commencing operation
krb5kdc: starting...
W20260924 07:13:25.284325 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:13:25 dist-test-slave-14ww krb5kdc[17338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:13:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:13:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:13:25.403501 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44167
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44167
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:25.482023 17351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:25.482230 17351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:25.482252 17351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:25.483399 17351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:13:25.483439 17351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:25.483453 17351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:25.483493 17351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:13:25.483525 17351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:13:25.484970 17351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:25.485216 17351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:25.485440 17351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:25.490975 17359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:25.490949 17357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:25.490991 17351 server_base.cc:1034] running on GCE node
W20260924 07:13:25.490957 17356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:25.491451 17351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:25.491683 17351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:25.492818 17351 hybrid_clock.cc:648] HybridClock initialized: now 1790234005492796 us; error 36 us; skew 500 ppm
Sep 24 07:13:25 dist-test-slave-14ww krb5kdc[17338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:25.495790 17351 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:13:25.496256 17351 webserver.cc:460] Webserver started at http://127.0.98.62:44701/ using document root <none> and password file <none>
I20260924 07:13:25.496475 17351 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:25.496543 17351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:25.496690 17351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:25.497720 17351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "7d17f017fa3d4ec0bb5b8b9b075bbe39"
format_stamp: "Formatted at 2026-09-24 07:13:25 on dist-test-slave-14ww"
server_key: "c10f23bcddaa0072d300aa8ad54ae6a0"
server_key_iv: "ca7dbaa147eb6a78a1b0e2f84d873834"
server_key_version: "encryptionkey@0"
I20260924 07:13:25.498077 17351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "7d17f017fa3d4ec0bb5b8b9b075bbe39"
format_stamp: "Formatted at 2026-09-24 07:13:25 on dist-test-slave-14ww"
server_key: "c10f23bcddaa0072d300aa8ad54ae6a0"
server_key_iv: "ca7dbaa147eb6a78a1b0e2f84d873834"
server_key_version: "encryptionkey@0"
I20260924 07:13:25.499356 17351 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 07:13:25.500021 17366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:25.500187 17351 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:13:25.500283 17351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "7d17f017fa3d4ec0bb5b8b9b075bbe39"
format_stamp: "Formatted at 2026-09-24 07:13:25 on dist-test-slave-14ww"
server_key: "c10f23bcddaa0072d300aa8ad54ae6a0"
server_key_iv: "ca7dbaa147eb6a78a1b0e2f84d873834"
server_key_version: "encryptionkey@0"
I20260924 07:13:25.500367 17351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:25.508121 17351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:25.511080 17351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:25.511230 17351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:25.515089 17351 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44167
I20260924 07:13:25.515141 17418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44167 every 8 connection(s)
I20260924 07:13:25.515468 17351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:13:25.516253 17419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:25.517462 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 17351
I20260924 07:13:25.517580 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:13:25.517774 392 external_mini_cluster.cc:1428] Setting key eb250996f7802a58f92a80a0ff60cc8a
I20260924 07:13:25.518957 17419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d17f017fa3d4ec0bb5b8b9b075bbe39: Bootstrap starting.
I20260924 07:13:25.519685 17419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d17f017fa3d4ec0bb5b8b9b075bbe39: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:25.519990 17419 log.cc:824] T 00000000000000000000000000000000 P 7d17f017fa3d4ec0bb5b8b9b075bbe39: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:25.520679 17419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d17f017fa3d4ec0bb5b8b9b075bbe39: No bootstrap required, opened a new log
I20260924 07:13:25.523113 17419 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d17f017fa3d4ec0bb5b8b9b075bbe39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d17f017fa3d4ec0bb5b8b9b075bbe39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44167 } }
I20260924 07:13:25.523307 17419 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d17f017fa3d4ec0bb5b8b9b075bbe39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:25.523344 17419 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d17f017fa3d4ec0bb5b8b9b075bbe39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d17f017fa3d4ec0bb5b8b9b075bbe39, State: Initialized, Role: FOLLOWER
I20260924 07:13:25.523468 17419 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d17f017fa3d4ec0bb5b8b9b075bbe39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d17f017fa3d4ec0bb5b8b9b075bbe39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44167 } }
I20260924 07:13:25.523545 17419 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d17f017fa3d4ec0bb5b8b9b075bbe39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:13:25.523602 17419 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d17f017fa3d4ec0bb5b8b9b075bbe39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:13:25.523656 17419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d17f017fa3d4ec0bb5b8b9b075bbe39 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:25.524385 17419 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d17f017fa3d4ec0bb5b8b9b075bbe39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d17f017fa3d4ec0bb5b8b9b075bbe39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44167 } }
I20260924 07:13:25.524508 17419 leader_election.cc:302] T 00000000000000000000000000000000 P 7d17f017fa3d4ec0bb5b8b9b075bbe39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d17f017fa3d4ec0bb5b8b9b075bbe39; no voters:
I20260924 07:13:25.524663 17419 leader_election.cc:288] T 00000000000000000000000000000000 P 7d17f017fa3d4ec0bb5b8b9b075bbe39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:13:25.524721 17424 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d17f017fa3d4ec0bb5b8b9b075bbe39 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:13:25.524891 17424 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d17f017fa3d4ec0bb5b8b9b075bbe39 [term 1 LEADER]: Becoming Leader. State: Replica: 7d17f017fa3d4ec0bb5b8b9b075bbe39, State: Running, Role: LEADER
I20260924 07:13:25.524942 17419 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d17f017fa3d4ec0bb5b8b9b075bbe39 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:13:25.525167 17424 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d17f017fa3d4ec0bb5b8b9b075bbe39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d17f017fa3d4ec0bb5b8b9b075bbe39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44167 } }
Sep 24 07:13:25 dist-test-slave-14ww krb5kdc[17338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:13:25.525527 17426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d17f017fa3d4ec0bb5b8b9b075bbe39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d17f017fa3d4ec0bb5b8b9b075bbe39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d17f017fa3d4ec0bb5b8b9b075bbe39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44167 } } }
I20260924 07:13:25.525602 17426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d17f017fa3d4ec0bb5b8b9b075bbe39 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:25.525746 17426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d17f017fa3d4ec0bb5b8b9b075bbe39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d17f017fa3d4ec0bb5b8b9b075bbe39. Latest consensus state: current_term: 1 leader_uuid: "7d17f017fa3d4ec0bb5b8b9b075bbe39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d17f017fa3d4ec0bb5b8b9b075bbe39" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44167 } } }
I20260924 07:13:25.525818 17433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:13:25.525884 17426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d17f017fa3d4ec0bb5b8b9b075bbe39 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:25.526429 17433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:13:25.528591 17433 catalog_manager.cc:1347] Generated new cluster ID: 3430fa1cea32454e8ad1acfbca99de89
I20260924 07:13:25.528684 17433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:13:25.530910 17422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:25.518276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40212 (local address 127.0.98.62:44167)
0924 07:13:25.518441 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:25.518448 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:25.518481 (+ 33us) server_negotiation.cc:407] Connection header received
0924 07:13:25.518864 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:25.518870 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:25.518910 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:25.519050 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:25.519848 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:25.521019 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:25.522067 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:25.522439 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:25.522516 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:25.526178 (+ 3662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:25.526182 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:25.526182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:25.526193 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:25.529383 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:25.529751 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:25.529755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:25.529755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:25.529854 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:25.530073 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:25.530073 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:25.530074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:25.530296 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:25.530459 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:25.530576 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 07:13:25.530712 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260924 07:13:25.543358 17433 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:13:25.543840 17433 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:13:25.553462 17433 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d17f017fa3d4ec0bb5b8b9b075bbe39: Generated new TSK 0
I20260924 07:13:25.553642 17433 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 07:13:25.584473 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44167
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:25.682417 17445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:25.682615 17445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:25.682638 17445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:25.683780 17445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:25.683821 17445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:25.683848 17445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:13:25.685281 17445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:25.685448 17445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:25.685611 17445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:25.690423 17451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:25.690440 17450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:25.690590 17445 server_base.cc:1034] running on GCE node
W20260924 07:13:25.690747 17453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:25.690972 17445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:25.691159 17445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:25.692315 17445 hybrid_clock.cc:648] HybridClock initialized: now 1790234005692278 us; error 51 us; skew 500 ppm
Sep 24 07:13:25 dist-test-slave-14ww krb5kdc[17338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:25.695462 17445 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:13:25.695827 17445 webserver.cc:460] Webserver started at http://127.0.98.1:36213/ using document root <none> and password file <none>
I20260924 07:13:25.696054 17445 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:25.696123 17445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:25.696273 17445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:25.697333 17445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-0/data/instance:
uuid: "b471ce4a95724f069ed69fe0952b7997"
format_stamp: "Formatted at 2026-09-24 07:13:25 on dist-test-slave-14ww"
server_key: "9b4dfbf70bb6b211242f2622a4e56e6c"
server_key_iv: "7c3ef15a860e8928f6282dcea4b3434c"
server_key_version: "encryptionkey@0"
I20260924 07:13:25.697687 17445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b471ce4a95724f069ed69fe0952b7997"
format_stamp: "Formatted at 2026-09-24 07:13:25 on dist-test-slave-14ww"
server_key: "9b4dfbf70bb6b211242f2622a4e56e6c"
server_key_iv: "7c3ef15a860e8928f6282dcea4b3434c"
server_key_version: "encryptionkey@0"
I20260924 07:13:25.699044 17445 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:13:25.699774 17460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:25.699954 17445 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:13:25.700062 17445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-0/wal
uuid: "b471ce4a95724f069ed69fe0952b7997"
format_stamp: "Formatted at 2026-09-24 07:13:25 on dist-test-slave-14ww"
server_key: "9b4dfbf70bb6b211242f2622a4e56e6c"
server_key_iv: "7c3ef15a860e8928f6282dcea4b3434c"
server_key_version: "encryptionkey@0"
I20260924 07:13:25.700146 17445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:25.711179 17445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:25.714224 17445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:25.714376 17445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:25.714583 17445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:13:25.714871 17445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:13:25.714926 17445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:25.714995 17445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:13:25.715040 17445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:25.720567 17445 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39805
I20260924 07:13:25.720636 17573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39805 every 8 connection(s)
I20260924 07:13:25.721020 17445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 07:13:25 dist-test-slave-14ww krb5kdc[17338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:13:25.729460 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 17445
I20260924 07:13:25.729594 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:13:25.729825 392 external_mini_cluster.cc:1428] Setting key b167d1dd219c983b0e050c088ecf4446
I20260924 07:13:25.732334 17422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:25.721802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54115 (local address 127.0.98.62:44167)
0924 07:13:25.721883 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:25.721886 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:25.722013 (+ 127us) server_negotiation.cc:407] Connection header received
0924 07:13:25.722450 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:25.722451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:25.722462 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:25.722510 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:25.723357 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:25.723866 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:25.725012 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:25.725241 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:25.725297 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:25.728143 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:25.728145 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:25.728145 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:25.728152 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:25.730901 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:25.731324 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:25.731326 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:25.731326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:25.731395 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:25.731656 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:25.731657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:25.731657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:25.731780 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:25.731857 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:25.732148 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 07:13:25.732246 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 07:13:25.732510 17576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:25.721718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44167 (local address 127.0.98.1:54115)
0924 07:13:25.721862 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:25.721932 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:25.722367 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:25.722568 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:25.722571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:25.722803 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:25.723276 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:25.723280 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:25.723969 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:25.723970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:25.724823 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:25.724826 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:25.725314 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:25.725315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:25.725414 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:25.726732 (+ 1318us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:25.726740 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:25.728057 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:25.731000 (+ 2943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:25.731008 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:25.731010 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:25.731239 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:25.731463 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:25.731464 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:25.731464 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:25.731585 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:25.731848 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:25.731854 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:25.732007 (+ 153us) client_negotiation.cc:769] Sending connection context
0924 07:13:25.732122 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 07:13:25.732277 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260924 07:13:25.733042 17574 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44167
I20260924 07:13:25.733160 17574 heartbeater.cc:464] Registering TS with master...
I20260924 07:13:25.733328 17574 heartbeater.cc:511] Master 127.0.98.62:44167 requested a full tablet report, sending...
I20260924 07:13:25.733889 17383 ts_manager.cc:194] Registered new tserver with Master: b471ce4a95724f069ed69fe0952b7997 (127.0.98.1:39805)
I20260924 07:13:25.734516 17383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54115
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 07:13:25.756538 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44167
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:25.836537 17579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:25.836773 17579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:25.836817 17579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:25.838037 17579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:25.838096 17579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:25.838140 17579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:13:25.839632 17579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:25.839882 17579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:25.840113 17579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:25.845130 17584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:25.845201 17585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:25.845204 17587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:25.845314 17579 server_base.cc:1034] running on GCE node
I20260924 07:13:25.845537 17579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:25.845773 17579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:25.846938 17579 hybrid_clock.cc:648] HybridClock initialized: now 1790234005846924 us; error 31 us; skew 500 ppm
Sep 24 07:13:25 dist-test-slave-14ww krb5kdc[17338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:25.850263 17579 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:13:25.850732 17579 webserver.cc:460] Webserver started at http://127.0.98.2:38381/ using document root <none> and password file <none>
I20260924 07:13:25.850963 17579 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:25.851030 17579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:25.851159 17579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:25.852139 17579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-1/data/instance:
uuid: "e575bd0e75dd4853ae13084b62bb13e5"
format_stamp: "Formatted at 2026-09-24 07:13:25 on dist-test-slave-14ww"
server_key: "4e72190212a10f0634fa98f351eb0703"
server_key_iv: "40de62560e6376e1beb105685a27d6c4"
server_key_version: "encryptionkey@0"
I20260924 07:13:25.852494 17579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e575bd0e75dd4853ae13084b62bb13e5"
format_stamp: "Formatted at 2026-09-24 07:13:25 on dist-test-slave-14ww"
server_key: "4e72190212a10f0634fa98f351eb0703"
server_key_iv: "40de62560e6376e1beb105685a27d6c4"
server_key_version: "encryptionkey@0"
I20260924 07:13:25.853850 17579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:13:25.854569 17594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:25.854794 17579 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:13:25.854892 17579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-1/wal
uuid: "e575bd0e75dd4853ae13084b62bb13e5"
format_stamp: "Formatted at 2026-09-24 07:13:25 on dist-test-slave-14ww"
server_key: "4e72190212a10f0634fa98f351eb0703"
server_key_iv: "40de62560e6376e1beb105685a27d6c4"
server_key_version: "encryptionkey@0"
I20260924 07:13:25.854979 17579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:25.877003 17579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:25.881286 17579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:25.881455 17579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:25.881676 17579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:13:25.881966 17579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:13:25.882018 17579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:25.882058 17579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:13:25.882087 17579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:25.887915 17579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40165
I20260924 07:13:25.887971 17707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40165 every 8 connection(s)
I20260924 07:13:25.888252 17579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:13:25.891074 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 17579
I20260924 07:13:25.891191 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:13:25.891395 392 external_mini_cluster.cc:1428] Setting key 64583328388b252c1ed0b2d97bc12d29
Sep 24 07:13:25 dist-test-slave-14ww krb5kdc[17338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:13:25.899421 17422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:25.888882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57533 (local address 127.0.98.62:44167)
0924 07:13:25.888925 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:25.888927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:25.889123 (+ 196us) server_negotiation.cc:407] Connection header received
0924 07:13:25.889561 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:25.889563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:25.889574 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:25.889629 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:25.890454 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:25.890941 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:25.892371 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:25.892557 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:25.892609 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:25.895195 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:25.895197 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:25.895197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:25.895205 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:25.897998 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:25.898370 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:25.898371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:25.898371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:25.898439 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:25.898729 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:25.898730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:25.898731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:25.898875 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:25.898960 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:25.899230 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 07:13:25.899310 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 07:13:25.899439 17710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:25.888869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44167 (local address 127.0.98.2:57533)
0924 07:13:25.889002 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:25.889034 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:25.889479 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:25.889680 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:25.889684 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:25.889912 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:25.890378 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:25.890382 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:25.891027 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:25.891028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:25.892285 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:25.892288 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:25.892639 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:25.892640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:25.892806 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:25.893818 (+ 1012us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:25.893826 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:25.895104 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:25.898081 (+ 2977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:25.898086 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:25.898088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:25.898292 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:25.898531 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:25.898532 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:25.898533 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:25.898648 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:25.898964 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:25.898970 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:25.899089 (+ 119us) client_negotiation.cc:769] Sending connection context
0924 07:13:25.899176 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 07:13:25.899287 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":66,"threads_started":1}
I20260924 07:13:25.900223 17708 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44167
I20260924 07:13:25.900303 17708 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 07:13:25.900506 17708 heartbeater.cc:511] Master 127.0.98.62:44167 requested a full tablet report, sending...
I20260924 07:13:25.900919 17383 ts_manager.cc:194] Registered new tserver with Master: e575bd0e75dd4853ae13084b62bb13e5 (127.0.98.2:40165)
I20260924 07:13:25.901257 17383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57533
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 07:13:25.919940 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44167
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:26.010460 17713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:26.010646 17713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:26.010669 17713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:26.011797 17713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:26.011837 17713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:26.011863 17713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:13:26.013468 17713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44167
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:26.013700 17713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:26.013921 17713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:26.019021 17718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:26.019027 17719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:26.019027 17721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:26.019084 17713 server_base.cc:1034] running on GCE node
I20260924 07:13:26.019553 17713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:26.019789 17713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:26.020947 17713 hybrid_clock.cc:648] HybridClock initialized: now 1790234006020921 us; error 44 us; skew 500 ppm
Sep 24 07:13:26 dist-test-slave-14ww krb5kdc[17338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:26.024190 17713 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:13:26.024715 17713 webserver.cc:460] Webserver started at http://127.0.98.3:37865/ using document root <none> and password file <none>
I20260924 07:13:26.024984 17713 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:26.025053 17713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:26.025198 17713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:26.026630 17713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-2/data/instance:
uuid: "6c568576676a4d33af1ef811af6c66b4"
format_stamp: "Formatted at 2026-09-24 07:13:26 on dist-test-slave-14ww"
server_key: "aebd4cbb158816a7696d8f49c073856f"
server_key_iv: "4261f82a5ecb3848485222ee866432cc"
server_key_version: "encryptionkey@0"
I20260924 07:13:26.027076 17713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6c568576676a4d33af1ef811af6c66b4"
format_stamp: "Formatted at 2026-09-24 07:13:26 on dist-test-slave-14ww"
server_key: "aebd4cbb158816a7696d8f49c073856f"
server_key_iv: "4261f82a5ecb3848485222ee866432cc"
server_key_version: "encryptionkey@0"
I20260924 07:13:26.028992 17713 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 07:13:26.029701 17728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:26.029865 17713 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:13:26.029959 17713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-2/wal
uuid: "6c568576676a4d33af1ef811af6c66b4"
format_stamp: "Formatted at 2026-09-24 07:13:26 on dist-test-slave-14ww"
server_key: "aebd4cbb158816a7696d8f49c073856f"
server_key_iv: "4261f82a5ecb3848485222ee866432cc"
server_key_version: "encryptionkey@0"
I20260924 07:13:26.030045 17713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:26.049229 17713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:26.053725 17713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:26.053882 17713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:26.054104 17713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:13:26.054397 17713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:13:26.054450 17713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:26.054491 17713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:13:26.054519 17713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:26.060195 17713 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36209
I20260924 07:13:26.060263 17841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36209 every 8 connection(s)
I20260924 07:13:26.060572 17713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:13:26.064708 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 17713
I20260924 07:13:26.064849 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:13:26.065039 392 external_mini_cluster.cc:1428] Setting key 849766913fa23c8d4347a563ea59af45
Sep 24 07:13:26 dist-test-slave-14ww krb5kdc[17338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234006, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:13:26.072309 17422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:26.061376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42001 (local address 127.0.98.62:44167)
0924 07:13:26.061455 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:26.061457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:26.061692 (+ 235us) server_negotiation.cc:407] Connection header received
0924 07:13:26.062115 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:26.062116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:26.062126 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:26.062194 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:26.063044 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:26.063549 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:26.064568 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:26.064881 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:26.064970 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:26.067879 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:26.067880 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:26.067880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:26.067887 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:26.070869 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:26.071356 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:26.071358 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:26.071358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:26.071435 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:26.071707 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:26.071708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:26.071709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:26.071827 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:26.071920 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:26.072178 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 07:13:26.072242 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 07:13:26.072477 17844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:26.061337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44167 (local address 127.0.98.3:42001)
0924 07:13:26.061504 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:26.061544 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:26.062011 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:26.062246 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:26.062249 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:26.062479 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:26.062940 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:26.062945 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:26.063629 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:26.063630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:26.064457 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:26.064460 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:26.064983 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:26.064984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:26.065129 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:26.066333 (+ 1204us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:26.066341 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:26.067784 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:26.071003 (+ 3219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:26.071010 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:26.071011 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:26.071280 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:26.071502 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:26.071503 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:26.071503 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:26.071609 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:26.071915 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:26.071919 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:26.072064 (+ 145us) client_negotiation.cc:769] Sending connection context
0924 07:13:26.072147 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 07:13:26.072271 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":102,"threads_started":1}
I20260924 07:13:26.072988 17842 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44167
I20260924 07:13:26.073073 17842 heartbeater.cc:464] Registering TS with master...
I20260924 07:13:26.073275 17842 heartbeater.cc:511] Master 127.0.98.62:44167 requested a full tablet report, sending...
I20260924 07:13:26.073652 17383 ts_manager.cc:194] Registered new tserver with Master: 6c568576676a4d33af1ef811af6c66b4 (127.0.98.3:36209)
I20260924 07:13:26.074031 17383 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42001
I20260924 07:13:26.077003 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:13:26.085479 17422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:26.078361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40220 (local address 127.0.98.62:44167)
0924 07:13:26.078428 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:26.078430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:26.078534 (+ 104us) server_negotiation.cc:407] Connection header received
0924 07:13:26.078623 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:26.078624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:26.078634 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:26.078685 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:26.079224 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:26.079690 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:26.080491 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:26.080677 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:26.080768 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:26.082349 (+ 1581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:26.082351 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:26.082351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:26.082358 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:26.084294 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:26.084672 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:26.084674 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:26.084674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:26.084766 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:26.084961 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:26.084962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:26.084962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:26.085101 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:26.085205 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:26.085309 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 07:13:26.085382 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 07:13:26.088131 17383 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40220:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:13:26.089874 17383 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:13:26.096325 17854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:26.092211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40165 (local address 127.0.0.1:42506)
0924 07:13:26.092384 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:26.092554 (+ 170us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:26.092589 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:26.092866 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:26.092869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:26.092872 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:26.092946 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:26.092948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:26.093970 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:26.093971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:26.095196 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:26.095199 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:26.095991 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:26.095992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:26.096079 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:26.096092 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 07:13:26.096188 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 07:13:26.096242 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":10,"thread_start_us":77,"threads_started":1}
I20260924 07:13:26.096369 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:26.092454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42506 (local address 127.0.98.2:40165)
0924 07:13:26.092597 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:26.092602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:26.092613 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:13:26.092655 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:26.092657 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:26.092674 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:26.092904 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:26.093062 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:26.093885 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:26.095301 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:26.095903 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:26.095992 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:26.096050 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:26.096207 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 07:13:26.096249 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":74,"threads_started":1}
I20260924 07:13:26.096514 17853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:26.092211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36209 (local address 127.0.0.1:51042)
0924 07:13:26.092369 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:26.092407 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:26.092474 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:26.092903 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:26.092904 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:26.092907 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:26.092970 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:26.092972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:26.094302 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:26.094303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:26.095363 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:26.095364 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:26.096265 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:26.096266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:26.096347 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:26.096352 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:13:26.096382 (+ 30us) client_negotiation.cc:240] Negotiation successful
0924 07:13:26.096430 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":24,"thread_start_us":58,"threads_started":1}
I20260924 07:13:26.096518 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:26.092226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39805 (local address 127.0.0.1:45738)
0924 07:13:26.092443 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:26.092451 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:26.092490 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:26.093217 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:26.093218 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:26.093220 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:26.093256 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:26.093257 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:26.094471 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:26.094472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:26.095462 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:26.095465 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:26.096336 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:26.096337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:26.096394 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:26.096401 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 07:13:26.096429 (+ 28us) client_negotiation.cc:240] Negotiation successful
0924 07:13:26.096457 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":4,"thread_start_us":90,"threads_started":1}
I20260924 07:13:26.096859 17858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:26.092625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45738 (local address 127.0.98.1:39805)
0924 07:13:26.092847 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:26.092851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:26.092861 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:13:26.093027 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:26.093030 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:26.093111 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:26.093213 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:26.093331 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:26.094394 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:26.095551 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:26.096251 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:26.096560 (+ 309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:26.096631 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:26.096663 (+ 32us) server_negotiation.cc:299] Negotiation successful
0924 07:13:26.096718 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":88,"threads_started":1}
I20260924 07:13:26.097080 17642 tablet_service.cc:1467] Processing CreateTablet for tablet a53e8c840e6a4d3b820621acc558ef19 (DEFAULT_TABLE table=test-table [id=ecb7d1723af94f30a245be5948fd76d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:13:26.097482 17642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a53e8c840e6a4d3b820621acc558ef19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:26.097669 17508 tablet_service.cc:1467] Processing CreateTablet for tablet a53e8c840e6a4d3b820621acc558ef19 (DEFAULT_TABLE table=test-table [id=ecb7d1723af94f30a245be5948fd76d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:13:26.097970 17508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a53e8c840e6a4d3b820621acc558ef19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:26.100050 17859 tablet_bootstrap.cc:492] T a53e8c840e6a4d3b820621acc558ef19 P e575bd0e75dd4853ae13084b62bb13e5: Bootstrap starting.
I20260924 07:13:26.100567 17856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:26.092371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51042 (local address 127.0.98.3:36209)
0924 07:13:26.092695 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:26.092699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:26.092706 (+ 7us) server_negotiation.cc:407] Connection header received
0924 07:13:26.092779 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:26.092782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:26.092798 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:26.093058 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:26.093129 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:26.094213 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:26.095444 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:26.096155 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:26.100378 (+ 4223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:26.100440 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:26.100465 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 07:13:26.100511 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":78,"threads_started":1}
I20260924 07:13:26.100845 17859 tablet_bootstrap.cc:654] T a53e8c840e6a4d3b820621acc558ef19 P e575bd0e75dd4853ae13084b62bb13e5: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:26.100907 17860 tablet_bootstrap.cc:492] T a53e8c840e6a4d3b820621acc558ef19 P b471ce4a95724f069ed69fe0952b7997: Bootstrap starting.
I20260924 07:13:26.101352 17859 log.cc:824] T a53e8c840e6a4d3b820621acc558ef19 P e575bd0e75dd4853ae13084b62bb13e5: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:26.101266 17776 tablet_service.cc:1467] Processing CreateTablet for tablet a53e8c840e6a4d3b820621acc558ef19 (DEFAULT_TABLE table=test-table [id=ecb7d1723af94f30a245be5948fd76d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:13:26.101531 17860 tablet_bootstrap.cc:654] T a53e8c840e6a4d3b820621acc558ef19 P b471ce4a95724f069ed69fe0952b7997: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:26.101554 17776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a53e8c840e6a4d3b820621acc558ef19. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:26.101989 17859 tablet_bootstrap.cc:492] T a53e8c840e6a4d3b820621acc558ef19 P e575bd0e75dd4853ae13084b62bb13e5: No bootstrap required, opened a new log
I20260924 07:13:26.102033 17859 ts_tablet_manager.cc:1397] T a53e8c840e6a4d3b820621acc558ef19 P e575bd0e75dd4853ae13084b62bb13e5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 07:13:26.102082 17860 log.cc:824] T a53e8c840e6a4d3b820621acc558ef19 P b471ce4a95724f069ed69fe0952b7997: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:26.103868 17859 raft_consensus.cc:348] T a53e8c840e6a4d3b820621acc558ef19 P e575bd0e75dd4853ae13084b62bb13e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e575bd0e75dd4853ae13084b62bb13e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40165 } } peers { permanent_uuid: "6c568576676a4d33af1ef811af6c66b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36209 } } peers { permanent_uuid: "b471ce4a95724f069ed69fe0952b7997" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39805 } }
I20260924 07:13:26.104005 17859 raft_consensus.cc:374] T a53e8c840e6a4d3b820621acc558ef19 P e575bd0e75dd4853ae13084b62bb13e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:26.104027 17859 raft_consensus.cc:729] T a53e8c840e6a4d3b820621acc558ef19 P e575bd0e75dd4853ae13084b62bb13e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e575bd0e75dd4853ae13084b62bb13e5, State: Initialized, Role: FOLLOWER
I20260924 07:13:26.104110 17859 consensus_queue.cc:260] T a53e8c840e6a4d3b820621acc558ef19 P e575bd0e75dd4853ae13084b62bb13e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e575bd0e75dd4853ae13084b62bb13e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40165 } } peers { permanent_uuid: "6c568576676a4d33af1ef811af6c66b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36209 } } peers { permanent_uuid: "b471ce4a95724f069ed69fe0952b7997" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39805 } }
I20260924 07:13:26.104348 17859 ts_tablet_manager.cc:1428] T a53e8c840e6a4d3b820621acc558ef19 P e575bd0e75dd4853ae13084b62bb13e5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 07:13:26.104389 17708 heartbeater.cc:503] Master 127.0.98.62:44167 was elected leader, sending a full tablet report...
I20260924 07:13:26.104830 17864 tablet_bootstrap.cc:492] T a53e8c840e6a4d3b820621acc558ef19 P 6c568576676a4d33af1ef811af6c66b4: Bootstrap starting.
I20260924 07:13:26.105186 17860 tablet_bootstrap.cc:492] T a53e8c840e6a4d3b820621acc558ef19 P b471ce4a95724f069ed69fe0952b7997: No bootstrap required, opened a new log
I20260924 07:13:26.105275 17860 ts_tablet_manager.cc:1397] T a53e8c840e6a4d3b820621acc558ef19 P b471ce4a95724f069ed69fe0952b7997: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 07:13:26.105476 17864 tablet_bootstrap.cc:654] T a53e8c840e6a4d3b820621acc558ef19 P 6c568576676a4d33af1ef811af6c66b4: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:26.106057 17864 log.cc:824] T a53e8c840e6a4d3b820621acc558ef19 P 6c568576676a4d33af1ef811af6c66b4: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:26.106681 17864 tablet_bootstrap.cc:492] T a53e8c840e6a4d3b820621acc558ef19 P 6c568576676a4d33af1ef811af6c66b4: No bootstrap required, opened a new log
I20260924 07:13:26.106732 17864 ts_tablet_manager.cc:1397] T a53e8c840e6a4d3b820621acc558ef19 P 6c568576676a4d33af1ef811af6c66b4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:13:26.106969 17860 raft_consensus.cc:348] T a53e8c840e6a4d3b820621acc558ef19 P b471ce4a95724f069ed69fe0952b7997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e575bd0e75dd4853ae13084b62bb13e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40165 } } peers { permanent_uuid: "6c568576676a4d33af1ef811af6c66b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36209 } } peers { permanent_uuid: "b471ce4a95724f069ed69fe0952b7997" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39805 } }
I20260924 07:13:26.107113 17860 raft_consensus.cc:374] T a53e8c840e6a4d3b820621acc558ef19 P b471ce4a95724f069ed69fe0952b7997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:26.107155 17860 raft_consensus.cc:729] T a53e8c840e6a4d3b820621acc558ef19 P b471ce4a95724f069ed69fe0952b7997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b471ce4a95724f069ed69fe0952b7997, State: Initialized, Role: FOLLOWER
I20260924 07:13:26.107270 17860 consensus_queue.cc:260] T a53e8c840e6a4d3b820621acc558ef19 P b471ce4a95724f069ed69fe0952b7997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e575bd0e75dd4853ae13084b62bb13e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40165 } } peers { permanent_uuid: "6c568576676a4d33af1ef811af6c66b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36209 } } peers { permanent_uuid: "b471ce4a95724f069ed69fe0952b7997" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39805 } }
I20260924 07:13:26.107547 17860 ts_tablet_manager.cc:1428] T a53e8c840e6a4d3b820621acc558ef19 P b471ce4a95724f069ed69fe0952b7997: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 07:13:26.107707 17574 heartbeater.cc:503] Master 127.0.98.62:44167 was elected leader, sending a full tablet report...
I20260924 07:13:26.108361 17864 raft_consensus.cc:348] T a53e8c840e6a4d3b820621acc558ef19 P 6c568576676a4d33af1ef811af6c66b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e575bd0e75dd4853ae13084b62bb13e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40165 } } peers { permanent_uuid: "6c568576676a4d33af1ef811af6c66b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36209 } } peers { permanent_uuid: "b471ce4a95724f069ed69fe0952b7997" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39805 } }
I20260924 07:13:26.108479 17864 raft_consensus.cc:374] T a53e8c840e6a4d3b820621acc558ef19 P 6c568576676a4d33af1ef811af6c66b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:26.108501 17864 raft_consensus.cc:729] T a53e8c840e6a4d3b820621acc558ef19 P 6c568576676a4d33af1ef811af6c66b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c568576676a4d33af1ef811af6c66b4, State: Initialized, Role: FOLLOWER
I20260924 07:13:26.108598 17864 consensus_queue.cc:260] T a53e8c840e6a4d3b820621acc558ef19 P 6c568576676a4d33af1ef811af6c66b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e575bd0e75dd4853ae13084b62bb13e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40165 } } peers { permanent_uuid: "6c568576676a4d33af1ef811af6c66b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36209 } } peers { permanent_uuid: "b471ce4a95724f069ed69fe0952b7997" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39805 } }
I20260924 07:13:26.108901 17864 ts_tablet_manager.cc:1428] T a53e8c840e6a4d3b820621acc558ef19 P 6c568576676a4d33af1ef811af6c66b4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 07:13:26.108911 17842 heartbeater.cc:503] Master 127.0.98.62:44167 was elected leader, sending a full tablet report...
W20260924 07:13:26.138783 17709 tablet.cc:2369] T a53e8c840e6a4d3b820621acc558ef19 P e575bd0e75dd4853ae13084b62bb13e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:13:26.221779 17575 tablet.cc:2369] T a53e8c840e6a4d3b820621acc558ef19 P b471ce4a95724f069ed69fe0952b7997: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 07:13:26.311219 17843 tablet.cc:2369] T a53e8c840e6a4d3b820621acc558ef19 P 6c568576676a4d33af1ef811af6c66b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:13:26.479415 17866 raft_consensus.cc:482] T a53e8c840e6a4d3b820621acc558ef19 P b471ce4a95724f069ed69fe0952b7997 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:13:26.479586 17866 raft_consensus.cc:504] T a53e8c840e6a4d3b820621acc558ef19 P b471ce4a95724f069ed69fe0952b7997 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e575bd0e75dd4853ae13084b62bb13e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40165 } } peers { permanent_uuid: "6c568576676a4d33af1ef811af6c66b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36209 } } peers { permanent_uuid: "b471ce4a95724f069ed69fe0952b7997" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39805 } }
I20260924 07:13:26.480084 17866 leader_election.cc:288] T a53e8c840e6a4d3b820621acc558ef19 P b471ce4a95724f069ed69fe0952b7997 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e575bd0e75dd4853ae13084b62bb13e5 (127.0.98.2:40165), 6c568576676a4d33af1ef811af6c66b4 (127.0.98.3:36209)
I20260924 07:13:26.483088 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:26.480405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40807 (local address 127.0.98.2:40165)
0924 07:13:26.480486 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:26.480489 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:26.480503 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:13:26.480561 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:26.480562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:26.480574 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:26.480634 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:26.480846 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:26.481372 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:26.482367 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:26.482724 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:26.482785 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:26.482792 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:26.482970 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 07:13:26.483014 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 07:13:26.483131 17856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:26.480408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40571 (local address 127.0.98.3:36209)
0924 07:13:26.480456 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:26.480459 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:26.480634 (+ 175us) server_negotiation.cc:407] Connection header received
0924 07:13:26.480665 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:26.480666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:26.480675 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:26.480767 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:26.480957 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:26.481393 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:26.482371 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:26.482742 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:26.482834 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:26.482843 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:26.483020 (+ 177us) server_negotiation.cc:299] Negotiation successful
0924 07:13:26.483058 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 07:13:26.483141 17868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:26.480313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36209 (local address 127.0.98.1:40571)
0924 07:13:26.480509 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:26.480521 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:26.480607 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:26.480818 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:26.480819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:26.480821 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:26.480898 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:26.480900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:26.481505 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:26.481506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:26.482274 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:26.482276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:26.482849 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:26.482851 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:26.482936 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:26.482940 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:13:26.482999 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 07:13:26.483036 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":50,"threads_started":1}
I20260924 07:13:26.483160 17869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:26.480273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40165 (local address 127.0.98.1:40807)
0924 07:13:26.480426 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:26.480439 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:26.480499 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:26.480659 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:26.480660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:26.480662 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:26.480770 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:26.480772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:26.481450 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:26.481451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:26.482273 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:26.482275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:26.482814 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:26.482815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:26.482886 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:26.482891 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:13:26.482947 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 07:13:26.483007 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":4,"thread_start_us":65,"threads_started":1}
I20260924 07:13:26.483508 17796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53e8c840e6a4d3b820621acc558ef19" candidate_uuid: "b471ce4a95724f069ed69fe0952b7997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c568576676a4d33af1ef811af6c66b4" is_pre_election: true
I20260924 07:13:26.483544 17662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53e8c840e6a4d3b820621acc558ef19" candidate_uuid: "b471ce4a95724f069ed69fe0952b7997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e575bd0e75dd4853ae13084b62bb13e5" is_pre_election: true
I20260924 07:13:26.483665 17662 raft_consensus.cc:2454] T a53e8c840e6a4d3b820621acc558ef19 P e575bd0e75dd4853ae13084b62bb13e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b471ce4a95724f069ed69fe0952b7997 in term 0.
I20260924 07:13:26.483661 17796 raft_consensus.cc:2454] T a53e8c840e6a4d3b820621acc558ef19 P 6c568576676a4d33af1ef811af6c66b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b471ce4a95724f069ed69fe0952b7997 in term 0.
I20260924 07:13:26.483901 17462 leader_election.cc:302] T a53e8c840e6a4d3b820621acc558ef19 P b471ce4a95724f069ed69fe0952b7997 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c568576676a4d33af1ef811af6c66b4, b471ce4a95724f069ed69fe0952b7997; no voters:
I20260924 07:13:26.484051 17866 raft_consensus.cc:2781] T a53e8c840e6a4d3b820621acc558ef19 P b471ce4a95724f069ed69fe0952b7997 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:13:26.484105 17866 raft_consensus.cc:482] T a53e8c840e6a4d3b820621acc558ef19 P b471ce4a95724f069ed69fe0952b7997 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:13:26.484135 17866 raft_consensus.cc:3037] T a53e8c840e6a4d3b820621acc558ef19 P b471ce4a95724f069ed69fe0952b7997 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:26.484892 17866 raft_consensus.cc:504] T a53e8c840e6a4d3b820621acc558ef19 P b471ce4a95724f069ed69fe0952b7997 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e575bd0e75dd4853ae13084b62bb13e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40165 } } peers { permanent_uuid: "6c568576676a4d33af1ef811af6c66b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36209 } } peers { permanent_uuid: "b471ce4a95724f069ed69fe0952b7997" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39805 } }
I20260924 07:13:26.485039 17866 leader_election.cc:288] T a53e8c840e6a4d3b820621acc558ef19 P b471ce4a95724f069ed69fe0952b7997 [CANDIDATE]: Term 1 election: Requested vote from peers e575bd0e75dd4853ae13084b62bb13e5 (127.0.98.2:40165), 6c568576676a4d33af1ef811af6c66b4 (127.0.98.3:36209)
I20260924 07:13:26.485155 17662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53e8c840e6a4d3b820621acc558ef19" candidate_uuid: "b471ce4a95724f069ed69fe0952b7997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e575bd0e75dd4853ae13084b62bb13e5"
I20260924 07:13:26.485216 17796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53e8c840e6a4d3b820621acc558ef19" candidate_uuid: "b471ce4a95724f069ed69fe0952b7997" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c568576676a4d33af1ef811af6c66b4"
I20260924 07:13:26.485260 17662 raft_consensus.cc:3037] T a53e8c840e6a4d3b820621acc558ef19 P e575bd0e75dd4853ae13084b62bb13e5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:26.485296 17796 raft_consensus.cc:3037] T a53e8c840e6a4d3b820621acc558ef19 P 6c568576676a4d33af1ef811af6c66b4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:26.485950 17796 raft_consensus.cc:2454] T a53e8c840e6a4d3b820621acc558ef19 P 6c568576676a4d33af1ef811af6c66b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b471ce4a95724f069ed69fe0952b7997 in term 1.
I20260924 07:13:26.485982 17662 raft_consensus.cc:2454] T a53e8c840e6a4d3b820621acc558ef19 P e575bd0e75dd4853ae13084b62bb13e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b471ce4a95724f069ed69fe0952b7997 in term 1.
I20260924 07:13:26.486140 17462 leader_election.cc:302] T a53e8c840e6a4d3b820621acc558ef19 P b471ce4a95724f069ed69fe0952b7997 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c568576676a4d33af1ef811af6c66b4, b471ce4a95724f069ed69fe0952b7997; no voters:
I20260924 07:13:26.486243 17866 raft_consensus.cc:2781] T a53e8c840e6a4d3b820621acc558ef19 P b471ce4a95724f069ed69fe0952b7997 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:13:26.486428 17866 raft_consensus.cc:686] T a53e8c840e6a4d3b820621acc558ef19 P b471ce4a95724f069ed69fe0952b7997 [term 1 LEADER]: Becoming Leader. State: Replica: b471ce4a95724f069ed69fe0952b7997, State: Running, Role: LEADER
I20260924 07:13:26.486536 17866 consensus_queue.cc:237] T a53e8c840e6a4d3b820621acc558ef19 P b471ce4a95724f069ed69fe0952b7997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e575bd0e75dd4853ae13084b62bb13e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40165 } } peers { permanent_uuid: "6c568576676a4d33af1ef811af6c66b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36209 } } peers { permanent_uuid: "b471ce4a95724f069ed69fe0952b7997" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39805 } }
I20260924 07:13:26.487370 17383 catalog_manager.cc:5495] T a53e8c840e6a4d3b820621acc558ef19 P b471ce4a95724f069ed69fe0952b7997 reported cstate change: term changed from 0 to 1, leader changed from <none> to b471ce4a95724f069ed69fe0952b7997 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b471ce4a95724f069ed69fe0952b7997" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e575bd0e75dd4853ae13084b62bb13e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c568576676a4d33af1ef811af6c66b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b471ce4a95724f069ed69fe0952b7997" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39805 } health_report { overall_health: HEALTHY } } }
I20260924 07:13:26.535328 17858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:26.532789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45742 (local address 127.0.98.1:39805)
0924 07:13:26.532845 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:26.532847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:26.532857 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:13:26.532949 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:26.532950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:26.532982 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:26.533067 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:13:26.533206 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:26.533737 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:26.534518 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:26.534726 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:26.534804 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:26.534981 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:13:26.535084 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:13:26.535179 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:26.535221 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 07:13:26.535242 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 07:13:26.537153 17796 raft_consensus.cc:1261] T a53e8c840e6a4d3b820621acc558ef19 P 6c568576676a4d33af1ef811af6c66b4 [term 1 FOLLOWER]: Refusing update from remote peer b471ce4a95724f069ed69fe0952b7997: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:13:26.537175 17662 raft_consensus.cc:1261] T a53e8c840e6a4d3b820621acc558ef19 P e575bd0e75dd4853ae13084b62bb13e5 [term 1 FOLLOWER]: Refusing update from remote peer b471ce4a95724f069ed69fe0952b7997: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:13:26.537595 17870 consensus_queue.cc:1035] T a53e8c840e6a4d3b820621acc558ef19 P b471ce4a95724f069ed69fe0952b7997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c568576676a4d33af1ef811af6c66b4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:13:26.537765 17866 consensus_queue.cc:1035] T a53e8c840e6a4d3b820621acc558ef19 P b471ce4a95724f069ed69fe0952b7997 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e575bd0e75dd4853ae13084b62bb13e5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:13:26.540185 17873 mvcc.cc:205] Tried to move back new op lower bound from 7332798490773196800 to 7332798490569351168. Current Snapshot: MvccSnapshot[applied={T|T < 7332798490773196800}]
I20260924 07:13:26.540584 17875 mvcc.cc:205] Tried to move back new op lower bound from 7332798490773196800 to 7332798490569351168. Current Snapshot: MvccSnapshot[applied={T|T < 7332798490773196800}]
I20260924 07:13:26.542241 17382 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:13:26.542336 17382 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40220:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:13:26.543118 17382 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7d17f017fa3d4ec0bb5b8b9b075bbe39: Sending DeleteTablet for 3 replicas of tablet a53e8c840e6a4d3b820621acc558ef19
I20260924 07:13:26.543445 17776 tablet_service.cc:1514] Processing DeleteTablet for tablet a53e8c840e6a4d3b820621acc558ef19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:13:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51042
I20260924 07:13:26.543546 17508 tablet_service.cc:1514] Processing DeleteTablet for tablet a53e8c840e6a4d3b820621acc558ef19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:13:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45738
I20260924 07:13:26.543545 17642 tablet_service.cc:1514] Processing DeleteTablet for tablet a53e8c840e6a4d3b820621acc558ef19 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:13:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42506
I20260924 07:13:26.543718 17883 tablet_replica.cc:330] T a53e8c840e6a4d3b820621acc558ef19 P 6c568576676a4d33af1ef811af6c66b4: stopping tablet replica
I20260924 07:13:26.543823 17883 raft_consensus.cc:2229] T a53e8c840e6a4d3b820621acc558ef19 P 6c568576676a4d33af1ef811af6c66b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:13:26.543865 17885 tablet_replica.cc:330] T a53e8c840e6a4d3b820621acc558ef19 P b471ce4a95724f069ed69fe0952b7997: stopping tablet replica
I20260924 07:13:26.543977 17883 raft_consensus.cc:2258] T a53e8c840e6a4d3b820621acc558ef19 P 6c568576676a4d33af1ef811af6c66b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:13:26.543987 17885 raft_consensus.cc:2229] T a53e8c840e6a4d3b820621acc558ef19 P b471ce4a95724f069ed69fe0952b7997 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:13:26.544075 17885 raft_consensus.cc:2258] T a53e8c840e6a4d3b820621acc558ef19 P b471ce4a95724f069ed69fe0952b7997 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:13:26.544082 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 17445
I20260924 07:13:26.544893 17883 ts_tablet_manager.cc:1905] T a53e8c840e6a4d3b820621acc558ef19 P 6c568576676a4d33af1ef811af6c66b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:13:26.547072 17883 ts_tablet_manager.cc:1918] T a53e8c840e6a4d3b820621acc558ef19 P 6c568576676a4d33af1ef811af6c66b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:13:26.547153 17883 log.cc:1196] T a53e8c840e6a4d3b820621acc558ef19 P 6c568576676a4d33af1ef811af6c66b4: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790233772407728-392-0/minicluster-data/ts-2/wal/wals/a53e8c840e6a4d3b820621acc558ef19
I20260924 07:13:26.547386 17883 ts_tablet_manager.cc:1939] T a53e8c840e6a4d3b820621acc558ef19 P 6c568576676a4d33af1ef811af6c66b4: Deleting consensus metadata
I20260924 07:13:26.547782 17368 catalog_manager.cc:4841] TS 6c568576676a4d33af1ef811af6c66b4 (127.0.98.3:36209): tablet a53e8c840e6a4d3b820621acc558ef19 (table test-table [id=ecb7d1723af94f30a245be5948fd76d3]) successfully deleted
I20260924 07:13:26.548872 17884 tablet_replica.cc:330] T a53e8c840e6a4d3b820621acc558ef19 P e575bd0e75dd4853ae13084b62bb13e5: stopping tablet replica
I20260924 07:13:26.549012 17884 raft_consensus.cc:2229] T a53e8c840e6a4d3b820621acc558ef19 P e575bd0e75dd4853ae13084b62bb13e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:13:26.549187 17884 raft_consensus.cc:2258] T a53e8c840e6a4d3b820621acc558ef19 P e575bd0e75dd4853ae13084b62bb13e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:13:26.549549 17884 ts_tablet_manager.cc:1905] T a53e8c840e6a4d3b820621acc558ef19 P e575bd0e75dd4853ae13084b62bb13e5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 07:13:26.550352 17370 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39805 (error 108)
I20260924 07:13:26.550588 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 17579
I20260924 07:13:26.551425 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:26.551299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39805 (local address 127.0.0.1:45748)
0924 07:13:26.551359 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:26.551395 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 07:13:26.551561 17370 catalog_manager.cc:4584] TS b471ce4a95724f069ed69fe0952b7997 (127.0.98.1:39805): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a53e8c840e6a4d3b820621acc558ef19: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39805: connect: Connection refused (error 111)
I20260924 07:13:26.555648 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:26.555558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40165 (local address 127.0.0.1:42518)
0924 07:13:26.555604 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:26.555628 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 07:13:26.555649 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 17713
I20260924 07:13:26.560621 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 17351
2026-09-24T07:13:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2313 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T07:13:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:13:26Z Disabled control of system clock
I20260924 07:13:26.587765 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46743
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 07:13:26.666776 17890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:26.666980 17890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:26.666999 17890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:26.668056 17890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:13:26.668088 17890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:26.668102 17890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:26.668113 17890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:13:26.668125 17890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:13:26.669378 17890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46743
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:26.669550 17890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:26.669760 17890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:26.674538 17895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:26.674559 17890 server_base.cc:1034] running on GCE node
W20260924 07:13:26.674643 17898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:26.674554 17896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:26.675043 17890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:26.675320 17890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:26.676496 17890 hybrid_clock.cc:648] HybridClock initialized: now 1790234006676486 us; error 46 us; skew 500 ppm
I20260924 07:13:26.678369 17890 webserver.cc:460] Webserver started at http://127.0.98.62:42695/ using document root <none> and password file <none>
I20260924 07:13:26.678618 17890 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:26.678689 17890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:26.678821 17890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:26.679847 17890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "1c6c78acfca343b987880f8f85cec6fe"
format_stamp: "Formatted at 2026-09-24 07:13:26 on dist-test-slave-14ww"
server_key: "62325e53c390dfd028c18706454ec7b0"
server_key_iv: "0d705559f04e09315d5a4c43c2f89cbc"
server_key_version: "encryptionkey@0"
I20260924 07:13:26.680220 17890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "1c6c78acfca343b987880f8f85cec6fe"
format_stamp: "Formatted at 2026-09-24 07:13:26 on dist-test-slave-14ww"
server_key: "62325e53c390dfd028c18706454ec7b0"
server_key_iv: "0d705559f04e09315d5a4c43c2f89cbc"
server_key_version: "encryptionkey@0"
I20260924 07:13:26.681785 17890 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:13:26.682633 17904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:26.682822 17890 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:13:26.682930 17890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "1c6c78acfca343b987880f8f85cec6fe"
format_stamp: "Formatted at 2026-09-24 07:13:26 on dist-test-slave-14ww"
server_key: "62325e53c390dfd028c18706454ec7b0"
server_key_iv: "0d705559f04e09315d5a4c43c2f89cbc"
server_key_version: "encryptionkey@0"
I20260924 07:13:26.683008 17890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:26.693614 17890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:26.693889 17890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:26.694033 17890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:26.697737 17890 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46743
I20260924 07:13:26.697790 17956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46743 every 8 connection(s)
I20260924 07:13:26.698127 17890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:13:26.699018 17957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:26.701258 17957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c6c78acfca343b987880f8f85cec6fe: Bootstrap starting.
I20260924 07:13:26.701678 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 17890
I20260924 07:13:26.701771 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:13:26.701881 17957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c6c78acfca343b987880f8f85cec6fe: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:26.702035 392 external_mini_cluster.cc:1428] Setting key 48187479e9baf5fa02ebad2c6f64ed9a
I20260924 07:13:26.702219 17957 log.cc:824] T 00000000000000000000000000000000 P 1c6c78acfca343b987880f8f85cec6fe: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:26.702966 17957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c6c78acfca343b987880f8f85cec6fe: No bootstrap required, opened a new log
I20260924 07:13:26.705502 17957 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c6c78acfca343b987880f8f85cec6fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c6c78acfca343b987880f8f85cec6fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46743 } }
I20260924 07:13:26.705628 17957 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c6c78acfca343b987880f8f85cec6fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:26.705649 17957 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c6c78acfca343b987880f8f85cec6fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c6c78acfca343b987880f8f85cec6fe, State: Initialized, Role: FOLLOWER
I20260924 07:13:26.705782 17957 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c6c78acfca343b987880f8f85cec6fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c6c78acfca343b987880f8f85cec6fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46743 } }
I20260924 07:13:26.705864 17957 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c6c78acfca343b987880f8f85cec6fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:13:26.705905 17957 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c6c78acfca343b987880f8f85cec6fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:13:26.705955 17957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c6c78acfca343b987880f8f85cec6fe [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:26.706596 17957 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c6c78acfca343b987880f8f85cec6fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c6c78acfca343b987880f8f85cec6fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46743 } }
I20260924 07:13:26.706698 17957 leader_election.cc:302] T 00000000000000000000000000000000 P 1c6c78acfca343b987880f8f85cec6fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c6c78acfca343b987880f8f85cec6fe; no voters:
I20260924 07:13:26.706866 17957 leader_election.cc:288] T 00000000000000000000000000000000 P 1c6c78acfca343b987880f8f85cec6fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:13:26.706928 17962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c6c78acfca343b987880f8f85cec6fe [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:13:26.707108 17957 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c6c78acfca343b987880f8f85cec6fe [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:13:26.707125 17962 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c6c78acfca343b987880f8f85cec6fe [term 1 LEADER]: Becoming Leader. State: Replica: 1c6c78acfca343b987880f8f85cec6fe, State: Running, Role: LEADER
I20260924 07:13:26.707268 17962 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c6c78acfca343b987880f8f85cec6fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c6c78acfca343b987880f8f85cec6fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46743 } }
I20260924 07:13:26.707638 17962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c6c78acfca343b987880f8f85cec6fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c6c78acfca343b987880f8f85cec6fe. Latest consensus state: current_term: 1 leader_uuid: "1c6c78acfca343b987880f8f85cec6fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c6c78acfca343b987880f8f85cec6fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46743 } } }
I20260924 07:13:26.707803 17962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c6c78acfca343b987880f8f85cec6fe [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:26.707649 17963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c6c78acfca343b987880f8f85cec6fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c6c78acfca343b987880f8f85cec6fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c6c78acfca343b987880f8f85cec6fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46743 } } }
I20260924 07:13:26.708079 17963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c6c78acfca343b987880f8f85cec6fe [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:26.708212 17971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:13:26.708761 17971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:13:26.709695 17961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:26.702546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37502 (local address 127.0.98.62:46743)
0924 07:13:26.702722 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:26.702731 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:26.702762 (+ 31us) server_negotiation.cc:407] Connection header received
0924 07:13:26.703361 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:26.703370 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:26.703442 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:26.703585 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:26.703755 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:26.705195 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:26.706129 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:26.706755 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:26.706882 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:26.708778 (+ 1896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:26.708782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:26.708782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:13:26.708980 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:26.709023 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:13:26.709039 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:26.709105 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:26.709381 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 07:13:26.709477 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":56,"threads_started":1}
I20260924 07:13:26.710461 17971 catalog_manager.cc:1347] Generated new cluster ID: ec6706778f9d4ec88e4ca49faac5d894
I20260924 07:13:26.710510 17971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:13:26.752422 17971 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:13:26.753201 17971 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:13:26.758453 17971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c6c78acfca343b987880f8f85cec6fe: Generated new TSK 0
I20260924 07:13:26.758637 17971 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:13:26.760970 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46743
--builtin_ntp_servers=127.0.98.20:42375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 07:13:26.842748 17981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:26.842950 17981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:26.842979 17981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:26.844130 17981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:26.844170 17981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:26.844197 17981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:13:26.845597 17981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:26.845844 17981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:26.846163 17981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:26.851361 17986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:26.851401 17987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:26.851402 17989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:26.851516 17981 server_base.cc:1034] running on GCE node
I20260924 07:13:26.851783 17981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:26.851974 17981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:26.853120 17981 hybrid_clock.cc:648] HybridClock initialized: now 1790234006853104 us; error 26 us; skew 500 ppm
I20260924 07:13:26.855278 17981 webserver.cc:460] Webserver started at http://127.0.98.1:43745/ using document root <none> and password file <none>
I20260924 07:13:26.855561 17981 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:26.855610 17981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:26.855705 17981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:26.856714 17981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-0/data/instance:
uuid: "584487bc06a5401193d35b3915731633"
format_stamp: "Formatted at 2026-09-24 07:13:26 on dist-test-slave-14ww"
server_key: "e1dd154e7a13d559ddcbee1802fa04db"
server_key_iv: "7dd227648c1ffe61d7bc72a4d37b7513"
server_key_version: "encryptionkey@0"
I20260924 07:13:26.857095 17981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-0/wal/instance:
uuid: "584487bc06a5401193d35b3915731633"
format_stamp: "Formatted at 2026-09-24 07:13:26 on dist-test-slave-14ww"
server_key: "e1dd154e7a13d559ddcbee1802fa04db"
server_key_iv: "7dd227648c1ffe61d7bc72a4d37b7513"
server_key_version: "encryptionkey@0"
I20260924 07:13:26.858309 17981 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:13:26.859045 17995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:26.859249 17981 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:13:26.859346 17981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-0/wal
uuid: "584487bc06a5401193d35b3915731633"
format_stamp: "Formatted at 2026-09-24 07:13:26 on dist-test-slave-14ww"
server_key: "e1dd154e7a13d559ddcbee1802fa04db"
server_key_iv: "7dd227648c1ffe61d7bc72a4d37b7513"
server_key_version: "encryptionkey@0"
I20260924 07:13:26.859431 17981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:26.884991 17981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:26.885300 17981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:26.885450 17981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:26.885705 17981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:13:26.886085 17981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:13:26.886139 17981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:26.886183 17981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:13:26.886231 17981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:26.892061 17981 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43827
I20260924 07:13:26.892100 18108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43827 every 8 connection(s)
I20260924 07:13:26.892450 17981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:13:26.896107 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 17981
I20260924 07:13:26.896219 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:13:26.896457 392 external_mini_cluster.cc:1428] Setting key cbf73f645039ff73f7e1c43228d02ef1
I20260924 07:13:26.897802 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46743
--builtin_ntp_servers=127.0.98.20:42375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 07:13:26.898409 17961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:26.893230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56969 (local address 127.0.98.62:46743)
0924 07:13:26.893306 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:26.893309 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:26.893566 (+ 257us) server_negotiation.cc:407] Connection header received
0924 07:13:26.893981 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:26.893982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:26.893993 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:26.894050 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:26.894698 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:26.895280 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:26.896337 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:26.896651 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:26.896712 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:26.898184 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:26.898185 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:26.898185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:13:26.898189 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:26.898204 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:13:26.898208 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:26.898250 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:26.898331 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 07:13:26.898353 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 07:13:26.898698 18111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:26.893211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46743 (local address 127.0.98.1:56969)
0924 07:13:26.893399 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:26.893432 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:26.893892 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:26.894100 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:26.894104 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:26.894111 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:26.894599 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:26.894602 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:26.895360 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:26.895361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:26.896142 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:26.896144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:26.896756 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:26.896757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:26.896908 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:26.898033 (+ 1125us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:13:26.898033 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:26.898070 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:13:26.898096 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:26.898260 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:26.898264 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:26.898267 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:13:26.898327 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 07:13:26.898434 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":105,"threads_started":1}
I20260924 07:13:26.899274 18109 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46743
I20260924 07:13:26.899384 18109 heartbeater.cc:464] Registering TS with master...
I20260924 07:13:26.899565 18109 heartbeater.cc:511] Master 127.0.98.62:46743 requested a full tablet report, sending...
I20260924 07:13:26.900097 17921 ts_manager.cc:194] Registered new tserver with Master: 584487bc06a5401193d35b3915731633 (127.0.98.1:43827)
I20260924 07:13:26.900688 17921 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:56969
W20260924 07:13:26.977782 18112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:26.977962 18112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:26.977982 18112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:26.979008 18112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:26.979039 18112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:26.979063 18112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:13:26.980276 18112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:26.980448 18112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:26.980683 18112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:26.985801 18118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:26.985817 18117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:26.985862 18112 server_base.cc:1034] running on GCE node
W20260924 07:13:26.985800 18120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:26.986259 18112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:26.986492 18112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:26.987648 18112 hybrid_clock.cc:648] HybridClock initialized: now 1790234006987649 us; error 39 us; skew 500 ppm
I20260924 07:13:26.997150 18112 webserver.cc:460] Webserver started at http://127.0.98.2:34727/ using document root <none> and password file <none>
I20260924 07:13:26.997411 18112 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:26.997485 18112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:26.997625 18112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:26.998631 18112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-1/data/instance:
uuid: "6496c16c497e43eab9c00ac30cbf45fc"
format_stamp: "Formatted at 2026-09-24 07:13:26 on dist-test-slave-14ww"
server_key: "6a61687c5bbebc236f43b514ca613de3"
server_key_iv: "5c017ae72daaea79cd5b19ef57640083"
server_key_version: "encryptionkey@0"
I20260924 07:13:26.998977 18112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6496c16c497e43eab9c00ac30cbf45fc"
format_stamp: "Formatted at 2026-09-24 07:13:26 on dist-test-slave-14ww"
server_key: "6a61687c5bbebc236f43b514ca613de3"
server_key_iv: "5c017ae72daaea79cd5b19ef57640083"
server_key_version: "encryptionkey@0"
I20260924 07:13:27.000205 18112 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:13:27.001029 18126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:27.001217 18112 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:13:27.001308 18112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-1/wal
uuid: "6496c16c497e43eab9c00ac30cbf45fc"
format_stamp: "Formatted at 2026-09-24 07:13:26 on dist-test-slave-14ww"
server_key: "6a61687c5bbebc236f43b514ca613de3"
server_key_iv: "5c017ae72daaea79cd5b19ef57640083"
server_key_version: "encryptionkey@0"
I20260924 07:13:27.001400 18112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:27.011897 18112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:27.012157 18112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:27.012298 18112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:27.012511 18112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:13:27.012902 18112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:13:27.012955 18112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:27.012990 18112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:13:27.013018 18112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:27.018599 18112 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40235
I20260924 07:13:27.018644 18239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40235 every 8 connection(s)
I20260924 07:13:27.018942 18112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:13:27.023496 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 18112
I20260924 07:13:27.023623 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:13:27.023830 392 external_mini_cluster.cc:1428] Setting key 404b42567194960945699f3ee04b17c9
I20260924 07:13:27.024169 17961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:27.019640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47419 (local address 127.0.98.62:46743)
0924 07:13:27.019726 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:27.019729 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:27.019933 (+ 204us) server_negotiation.cc:407] Connection header received
0924 07:13:27.020457 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:27.020458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:27.020471 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:27.020527 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:27.021264 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:27.021771 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:27.022771 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:27.022966 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:27.023043 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:27.023888 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:27.023889 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:27.023890 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:13:27.023895 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:27.023909 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:13:27.023913 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:27.023985 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:27.024066 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 07:13:27.024093 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 07:13:27.024379 18242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:27.019620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46743 (local address 127.0.98.2:47419)
0924 07:13:27.019811 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:27.019843 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:27.020343 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:27.020593 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:27.020597 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:27.020606 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:27.021170 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:27.021176 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:27.021881 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:27.021889 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:27.022669 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:27.022671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:27.023049 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:27.023051 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:27.023187 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:27.023738 (+ 551us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:13:27.023738 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:27.023777 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:13:27.023799 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:27.023986 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:27.023992 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:27.023995 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:13:27.024067 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 07:13:27.024167 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":68,"threads_started":1}
I20260924 07:13:27.024874 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46743
--builtin_ntp_servers=127.0.98.20:42375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 07:13:27.025117 18240 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46743
I20260924 07:13:27.025206 18240 heartbeater.cc:464] Registering TS with master...
I20260924 07:13:27.025396 18240 heartbeater.cc:511] Master 127.0.98.62:46743 requested a full tablet report, sending...
I20260924 07:13:27.025738 17921 ts_manager.cc:194] Registered new tserver with Master: 6496c16c497e43eab9c00ac30cbf45fc (127.0.98.2:40235)
I20260924 07:13:27.026158 17921 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:47419
W20260924 07:13:27.105082 18243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:27.105263 18243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:27.105283 18243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:27.106396 18243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:27.106434 18243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:27.106505 18243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:13:27.107970 18243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:27.108206 18243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:27.108512 18243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:27.113696 18248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:27.113754 18243 server_base.cc:1034] running on GCE node
W20260924 07:13:27.113826 18249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:27.113705 18251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:27.114159 18243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:27.114344 18243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:27.115492 18243 hybrid_clock.cc:648] HybridClock initialized: now 1790234007115480 us; error 27 us; skew 500 ppm
I20260924 07:13:27.116719 18243 webserver.cc:460] Webserver started at http://127.0.98.3:45443/ using document root <none> and password file <none>
I20260924 07:13:27.117018 18243 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:27.117091 18243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:27.117240 18243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:27.118224 18243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-2/data/instance:
uuid: "c486c9c9aec64cdb8cba5fecfdfec2cf"
format_stamp: "Formatted at 2026-09-24 07:13:27 on dist-test-slave-14ww"
server_key: "2067a87e01c00c1af42811d1b69d75d4"
server_key_iv: "fe7fbe4053579a6a1aadff8eca988c01"
server_key_version: "encryptionkey@0"
I20260924 07:13:27.118564 18243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c486c9c9aec64cdb8cba5fecfdfec2cf"
format_stamp: "Formatted at 2026-09-24 07:13:27 on dist-test-slave-14ww"
server_key: "2067a87e01c00c1af42811d1b69d75d4"
server_key_iv: "fe7fbe4053579a6a1aadff8eca988c01"
server_key_version: "encryptionkey@0"
I20260924 07:13:27.119803 18243 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:13:27.120525 18257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:27.120688 18243 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:13:27.120813 18243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-2/wal
uuid: "c486c9c9aec64cdb8cba5fecfdfec2cf"
format_stamp: "Formatted at 2026-09-24 07:13:27 on dist-test-slave-14ww"
server_key: "2067a87e01c00c1af42811d1b69d75d4"
server_key_iv: "fe7fbe4053579a6a1aadff8eca988c01"
server_key_version: "encryptionkey@0"
I20260924 07:13:27.120893 18243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:27.136811 18243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:27.137106 18243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:27.137259 18243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:27.137483 18243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:13:27.137811 18243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:13:27.137866 18243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:27.137899 18243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:13:27.137926 18243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:27.143494 18243 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45743
I20260924 07:13:27.143586 18370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45743 every 8 connection(s)
I20260924 07:13:27.143906 18243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:13:27.149269 17961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:27.144726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54651 (local address 127.0.98.62:46743)
0924 07:13:27.144855 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:27.144857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:27.145075 (+ 218us) server_negotiation.cc:407] Connection header received
0924 07:13:27.145606 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:27.145607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:27.145620 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:27.145668 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:27.146351 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:27.146851 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:27.147844 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:27.148058 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:27.148137 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:27.148982 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:27.148983 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:27.148983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:13:27.148988 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:27.149000 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:13:27.149005 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:27.149056 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:27.149157 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 07:13:27.149191 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260924 07:13:27.149478 18373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:27.144613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46743 (local address 127.0.98.3:54651)
0924 07:13:27.144937 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:27.144977 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:27.145498 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:27.145746 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:27.145749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:27.145760 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:27.146253 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:27.146258 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:27.146955 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:27.146957 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:27.147748 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:27.147750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:27.148143 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:27.148144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:27.148283 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:27.148793 (+ 510us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:13:27.148794 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:27.148822 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:13:27.148843 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:27.149062 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:27.149066 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:27.149070 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:13:27.149158 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 07:13:27.149257 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":160,"threads_started":1}
I20260924 07:13:27.149748 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 18243
I20260924 07:13:27.149870 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:13:27.149928 18371 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46743
I20260924 07:13:27.150043 18371 heartbeater.cc:464] Registering TS with master...
I20260924 07:13:27.150104 392 external_mini_cluster.cc:1428] Setting key 0a4d82542bea2630de023bfb9cb75ffe
I20260924 07:13:27.150231 18371 heartbeater.cc:511] Master 127.0.98.62:46743 requested a full tablet report, sending...
I20260924 07:13:27.150632 17921 ts_manager.cc:194] Registered new tserver with Master: c486c9c9aec64cdb8cba5fecfdfec2cf (127.0.98.3:45743)
I20260924 07:13:27.151036 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:13:27.151100 17921 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:54651
I20260924 07:13:27.155354 17961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:27.152520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37508 (local address 127.0.98.62:46743)
0924 07:13:27.152558 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:27.152559 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:27.152724 (+ 165us) server_negotiation.cc:407] Connection header received
0924 07:13:27.152811 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:27.152812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:27.152820 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:27.152864 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:27.153039 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:27.153500 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:27.154272 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:27.154466 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:27.154548 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:27.155110 (+ 562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:27.155111 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:27.155111 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:13:27.155115 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:27.155126 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:13:27.155130 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:27.155200 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:27.155259 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 07:13:27.155277 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 07:13:27.156782 17921 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37508:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:13:27.158265 17921 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:13:27.164469 18380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:27.160393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41622 (local address 127.0.98.3:45743)
0924 07:13:27.161237 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:27.161240 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:27.161248 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:13:27.161279 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:27.161281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:27.161295 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:27.161364 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:27.161456 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:27.162221 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:27.163543 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:27.164063 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:27.164162 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:27.164249 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:27.164362 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 07:13:27.164398 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":810,"thread_start_us":85,"threads_started":1}
I20260924 07:13:27.164467 18379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:27.160301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45743 (local address 127.0.0.1:41622)
0924 07:13:27.160402 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:27.160443 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:27.160499 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:27.161361 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:27.161361 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:27.161364 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:27.161405 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:27.161406 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:27.162314 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:27.162316 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:27.163474 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:27.163477 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:27.164164 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:27.164165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:27.164251 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:27.164265 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 07:13:27.164347 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 07:13:27.164395 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":54,"threads_started":1}
I20260924 07:13:27.165287 18382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:27.160499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43827 (local address 127.0.0.1:54752)
0924 07:13:27.160737 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:27.160764 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:27.160830 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:27.161126 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:27.161129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:27.161131 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:27.161194 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:27.161196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:27.162180 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:27.162181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:27.163344 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:27.163346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:27.165032 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:27.165033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:27.165097 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:27.165101 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:13:27.165165 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 07:13:27.165196 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":23,"thread_start_us":70,"threads_started":1}
I20260924 07:13:27.165390 18381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:27.160407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40235 (local address 127.0.0.1:43082)
0924 07:13:27.160633 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:27.160644 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:27.160700 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:27.161136 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:27.161137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:27.161140 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:27.161209 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:27.161212 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:27.163436 (+ 2224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:27.163437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:27.164096 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:27.164098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:27.165049 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:27.165051 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:27.165116 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:27.165121 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:13:27.165153 (+ 32us) client_negotiation.cc:240] Negotiation successful
0924 07:13:27.165198 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":72,"threads_started":1}
I20260924 07:13:27.165570 18384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:27.160713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54752 (local address 127.0.98.1:43827)
0924 07:13:27.160931 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:27.160935 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:27.160943 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:13:27.160979 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:27.160980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:27.160992 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:27.161325 (+ 333us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:27.161383 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:27.162105 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:27.164210 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:27.164946 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:27.165022 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:27.165104 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:27.165181 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 07:13:27.165496 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":136,"threads_started":1}
I20260924 07:13:27.165712 18305 tablet_service.cc:1467] Processing CreateTablet for tablet 78eaf92538c8474da872d64a5b07c974 (DEFAULT_TABLE table=test-table [id=5f317c3a8f8449ce8a1d6182337818ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:13:27.166060 18305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78eaf92538c8474da872d64a5b07c974. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:27.166213 18043 tablet_service.cc:1467] Processing CreateTablet for tablet 78eaf92538c8474da872d64a5b07c974 (DEFAULT_TABLE table=test-table [id=5f317c3a8f8449ce8a1d6182337818ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:13:27.166524 18043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78eaf92538c8474da872d64a5b07c974. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:27.168808 18385 tablet_bootstrap.cc:492] T 78eaf92538c8474da872d64a5b07c974 P c486c9c9aec64cdb8cba5fecfdfec2cf: Bootstrap starting.
I20260924 07:13:27.169132 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:27.160604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43082 (local address 127.0.98.2:40235)
0924 07:13:27.160969 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:27.160973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:27.160983 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:13:27.161022 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:27.161024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:27.161038 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:27.161118 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:27.162194 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:27.163340 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:27.164191 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:27.164938 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:27.168839 (+ 3901us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:27.168942 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:27.169005 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 07:13:27.169054 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":74,"threads_started":1}
I20260924 07:13:27.169523 18385 tablet_bootstrap.cc:654] T 78eaf92538c8474da872d64a5b07c974 P c486c9c9aec64cdb8cba5fecfdfec2cf: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:27.169842 18385 log.cc:824] T 78eaf92538c8474da872d64a5b07c974 P c486c9c9aec64cdb8cba5fecfdfec2cf: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:27.169767 18174 tablet_service.cc:1467] Processing CreateTablet for tablet 78eaf92538c8474da872d64a5b07c974 (DEFAULT_TABLE table=test-table [id=5f317c3a8f8449ce8a1d6182337818ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:13:27.170081 18174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78eaf92538c8474da872d64a5b07c974. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:27.170184 18386 tablet_bootstrap.cc:492] T 78eaf92538c8474da872d64a5b07c974 P 584487bc06a5401193d35b3915731633: Bootstrap starting.
I20260924 07:13:27.170645 18385 tablet_bootstrap.cc:492] T 78eaf92538c8474da872d64a5b07c974 P c486c9c9aec64cdb8cba5fecfdfec2cf: No bootstrap required, opened a new log
I20260924 07:13:27.170732 18385 ts_tablet_manager.cc:1397] T 78eaf92538c8474da872d64a5b07c974 P c486c9c9aec64cdb8cba5fecfdfec2cf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:13:27.170818 18386 tablet_bootstrap.cc:654] T 78eaf92538c8474da872d64a5b07c974 P 584487bc06a5401193d35b3915731633: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:27.171128 18386 log.cc:824] T 78eaf92538c8474da872d64a5b07c974 P 584487bc06a5401193d35b3915731633: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:27.171970 18386 tablet_bootstrap.cc:492] T 78eaf92538c8474da872d64a5b07c974 P 584487bc06a5401193d35b3915731633: No bootstrap required, opened a new log
I20260924 07:13:27.172044 18386 ts_tablet_manager.cc:1397] T 78eaf92538c8474da872d64a5b07c974 P 584487bc06a5401193d35b3915731633: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:13:27.173198 18385 raft_consensus.cc:348] T 78eaf92538c8474da872d64a5b07c974 P c486c9c9aec64cdb8cba5fecfdfec2cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486c9c9aec64cdb8cba5fecfdfec2cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45743 } } peers { permanent_uuid: "584487bc06a5401193d35b3915731633" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43827 } } peers { permanent_uuid: "6496c16c497e43eab9c00ac30cbf45fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40235 } }
I20260924 07:13:27.173416 18385 raft_consensus.cc:374] T 78eaf92538c8474da872d64a5b07c974 P c486c9c9aec64cdb8cba5fecfdfec2cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:27.173491 18385 raft_consensus.cc:729] T 78eaf92538c8474da872d64a5b07c974 P c486c9c9aec64cdb8cba5fecfdfec2cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c486c9c9aec64cdb8cba5fecfdfec2cf, State: Initialized, Role: FOLLOWER
I20260924 07:13:27.173518 18389 tablet_bootstrap.cc:492] T 78eaf92538c8474da872d64a5b07c974 P 6496c16c497e43eab9c00ac30cbf45fc: Bootstrap starting.
I20260924 07:13:27.173657 18385 consensus_queue.cc:260] T 78eaf92538c8474da872d64a5b07c974 P c486c9c9aec64cdb8cba5fecfdfec2cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486c9c9aec64cdb8cba5fecfdfec2cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45743 } } peers { permanent_uuid: "584487bc06a5401193d35b3915731633" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43827 } } peers { permanent_uuid: "6496c16c497e43eab9c00ac30cbf45fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40235 } }
I20260924 07:13:27.173969 18385 ts_tablet_manager.cc:1428] T 78eaf92538c8474da872d64a5b07c974 P c486c9c9aec64cdb8cba5fecfdfec2cf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 07:13:27.174000 18371 heartbeater.cc:503] Master 127.0.98.62:46743 was elected leader, sending a full tablet report...
I20260924 07:13:27.174198 18389 tablet_bootstrap.cc:654] T 78eaf92538c8474da872d64a5b07c974 P 6496c16c497e43eab9c00ac30cbf45fc: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:27.174328 18386 raft_consensus.cc:348] T 78eaf92538c8474da872d64a5b07c974 P 584487bc06a5401193d35b3915731633 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486c9c9aec64cdb8cba5fecfdfec2cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45743 } } peers { permanent_uuid: "584487bc06a5401193d35b3915731633" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43827 } } peers { permanent_uuid: "6496c16c497e43eab9c00ac30cbf45fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40235 } }
I20260924 07:13:27.174508 18386 raft_consensus.cc:374] T 78eaf92538c8474da872d64a5b07c974 P 584487bc06a5401193d35b3915731633 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:27.174546 18386 raft_consensus.cc:729] T 78eaf92538c8474da872d64a5b07c974 P 584487bc06a5401193d35b3915731633 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584487bc06a5401193d35b3915731633, State: Initialized, Role: FOLLOWER
I20260924 07:13:27.174543 18389 log.cc:824] T 78eaf92538c8474da872d64a5b07c974 P 6496c16c497e43eab9c00ac30cbf45fc: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:27.174696 18386 consensus_queue.cc:260] T 78eaf92538c8474da872d64a5b07c974 P 584487bc06a5401193d35b3915731633 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486c9c9aec64cdb8cba5fecfdfec2cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45743 } } peers { permanent_uuid: "584487bc06a5401193d35b3915731633" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43827 } } peers { permanent_uuid: "6496c16c497e43eab9c00ac30cbf45fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40235 } }
I20260924 07:13:27.174966 18386 ts_tablet_manager.cc:1428] T 78eaf92538c8474da872d64a5b07c974 P 584487bc06a5401193d35b3915731633: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 07:13:27.175012 18109 heartbeater.cc:503] Master 127.0.98.62:46743 was elected leader, sending a full tablet report...
I20260924 07:13:27.175551 18389 tablet_bootstrap.cc:492] T 78eaf92538c8474da872d64a5b07c974 P 6496c16c497e43eab9c00ac30cbf45fc: No bootstrap required, opened a new log
I20260924 07:13:27.175619 18389 ts_tablet_manager.cc:1397] T 78eaf92538c8474da872d64a5b07c974 P 6496c16c497e43eab9c00ac30cbf45fc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:13:27.177578 18389 raft_consensus.cc:348] T 78eaf92538c8474da872d64a5b07c974 P 6496c16c497e43eab9c00ac30cbf45fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486c9c9aec64cdb8cba5fecfdfec2cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45743 } } peers { permanent_uuid: "584487bc06a5401193d35b3915731633" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43827 } } peers { permanent_uuid: "6496c16c497e43eab9c00ac30cbf45fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40235 } }
I20260924 07:13:27.177721 18389 raft_consensus.cc:374] T 78eaf92538c8474da872d64a5b07c974 P 6496c16c497e43eab9c00ac30cbf45fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:27.177765 18389 raft_consensus.cc:729] T 78eaf92538c8474da872d64a5b07c974 P 6496c16c497e43eab9c00ac30cbf45fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6496c16c497e43eab9c00ac30cbf45fc, State: Initialized, Role: FOLLOWER
I20260924 07:13:27.177873 18389 consensus_queue.cc:260] T 78eaf92538c8474da872d64a5b07c974 P 6496c16c497e43eab9c00ac30cbf45fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486c9c9aec64cdb8cba5fecfdfec2cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45743 } } peers { permanent_uuid: "584487bc06a5401193d35b3915731633" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43827 } } peers { permanent_uuid: "6496c16c497e43eab9c00ac30cbf45fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40235 } }
I20260924 07:13:27.178109 18389 ts_tablet_manager.cc:1428] T 78eaf92538c8474da872d64a5b07c974 P 6496c16c497e43eab9c00ac30cbf45fc: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 07:13:27.178164 18240 heartbeater.cc:503] Master 127.0.98.62:46743 was elected leader, sending a full tablet report...
W20260924 07:13:27.269515 18241 tablet.cc:2369] T 78eaf92538c8474da872d64a5b07c974 P 6496c16c497e43eab9c00ac30cbf45fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:13:27.316983 18393 raft_consensus.cc:482] T 78eaf92538c8474da872d64a5b07c974 P 6496c16c497e43eab9c00ac30cbf45fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:13:27.317116 18393 raft_consensus.cc:504] T 78eaf92538c8474da872d64a5b07c974 P 6496c16c497e43eab9c00ac30cbf45fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486c9c9aec64cdb8cba5fecfdfec2cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45743 } } peers { permanent_uuid: "584487bc06a5401193d35b3915731633" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43827 } } peers { permanent_uuid: "6496c16c497e43eab9c00ac30cbf45fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40235 } }
I20260924 07:13:27.317453 18393 leader_election.cc:288] T 78eaf92538c8474da872d64a5b07c974 P 6496c16c497e43eab9c00ac30cbf45fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c486c9c9aec64cdb8cba5fecfdfec2cf (127.0.98.3:45743), 584487bc06a5401193d35b3915731633 (127.0.98.1:43827)
I20260924 07:13:27.320370 18380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:27.317786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59379 (local address 127.0.98.3:45743)
0924 07:13:27.317893 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:27.317895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:27.317909 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:13:27.317942 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:27.317943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:27.317952 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:27.318036 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:27.318206 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:27.318679 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:27.319621 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:27.320013 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:27.320095 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:27.320106 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:27.320262 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 07:13:27.320297 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 07:13:27.320370 18242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:27.317693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45743 (local address 127.0.98.2:59379)
0924 07:13:27.317764 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:27.317779 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:27.317846 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:27.318054 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:27.318055 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:27.318057 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:27.318127 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:27.318129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:27.318766 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:27.318767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:27.319525 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:27.319527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:27.320109 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:27.320110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:27.320181 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:27.320185 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:13:27.320256 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 07:13:27.320306 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 07:13:27.320544 18384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:27.317919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49107 (local address 127.0.98.1:43827)
0924 07:13:27.317991 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:27.317993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:27.318072 (+ 79us) server_negotiation.cc:407] Connection header received
0924 07:13:27.318168 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:27.318169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:27.318177 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:27.318258 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:27.318446 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:27.318901 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:27.319826 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:27.320181 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:27.320270 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:27.320281 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:27.320445 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 07:13:27.320484 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 07:13:27.320552 18394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:27.317811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43827 (local address 127.0.98.2:49107)
0924 07:13:27.317993 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:27.318003 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:27.318085 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:27.318264 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:27.318265 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:27.318266 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:27.318348 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:27.318350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:27.318983 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:27.318984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:27.319715 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:27.319717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:27.320283 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:27.320285 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:27.320351 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:27.320354 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:13:27.320435 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 07:13:27.320474 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":125,"threads_started":1}
I20260924 07:13:27.320762 18325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78eaf92538c8474da872d64a5b07c974" candidate_uuid: "6496c16c497e43eab9c00ac30cbf45fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c486c9c9aec64cdb8cba5fecfdfec2cf" is_pre_election: true
I20260924 07:13:27.320931 18325 raft_consensus.cc:2454] T 78eaf92538c8474da872d64a5b07c974 P c486c9c9aec64cdb8cba5fecfdfec2cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6496c16c497e43eab9c00ac30cbf45fc in term 0.
I20260924 07:13:27.321158 18127 leader_election.cc:302] T 78eaf92538c8474da872d64a5b07c974 P 6496c16c497e43eab9c00ac30cbf45fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6496c16c497e43eab9c00ac30cbf45fc, c486c9c9aec64cdb8cba5fecfdfec2cf; no voters:
I20260924 07:13:27.321305 18393 raft_consensus.cc:2781] T 78eaf92538c8474da872d64a5b07c974 P 6496c16c497e43eab9c00ac30cbf45fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:13:27.321310 18063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78eaf92538c8474da872d64a5b07c974" candidate_uuid: "6496c16c497e43eab9c00ac30cbf45fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584487bc06a5401193d35b3915731633" is_pre_election: true
I20260924 07:13:27.321367 18393 raft_consensus.cc:482] T 78eaf92538c8474da872d64a5b07c974 P 6496c16c497e43eab9c00ac30cbf45fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:13:27.321396 18393 raft_consensus.cc:3037] T 78eaf92538c8474da872d64a5b07c974 P 6496c16c497e43eab9c00ac30cbf45fc [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:27.321465 18063 raft_consensus.cc:2454] T 78eaf92538c8474da872d64a5b07c974 P 584487bc06a5401193d35b3915731633 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6496c16c497e43eab9c00ac30cbf45fc in term 0.
I20260924 07:13:27.322180 18393 raft_consensus.cc:504] T 78eaf92538c8474da872d64a5b07c974 P 6496c16c497e43eab9c00ac30cbf45fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486c9c9aec64cdb8cba5fecfdfec2cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45743 } } peers { permanent_uuid: "584487bc06a5401193d35b3915731633" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43827 } } peers { permanent_uuid: "6496c16c497e43eab9c00ac30cbf45fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40235 } }
I20260924 07:13:27.322329 18393 leader_election.cc:288] T 78eaf92538c8474da872d64a5b07c974 P 6496c16c497e43eab9c00ac30cbf45fc [CANDIDATE]: Term 1 election: Requested vote from peers c486c9c9aec64cdb8cba5fecfdfec2cf (127.0.98.3:45743), 584487bc06a5401193d35b3915731633 (127.0.98.1:43827)
I20260924 07:13:27.322547 18063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78eaf92538c8474da872d64a5b07c974" candidate_uuid: "6496c16c497e43eab9c00ac30cbf45fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "584487bc06a5401193d35b3915731633"
I20260924 07:13:27.322557 18325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78eaf92538c8474da872d64a5b07c974" candidate_uuid: "6496c16c497e43eab9c00ac30cbf45fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c486c9c9aec64cdb8cba5fecfdfec2cf"
I20260924 07:13:27.322640 18325 raft_consensus.cc:3037] T 78eaf92538c8474da872d64a5b07c974 P c486c9c9aec64cdb8cba5fecfdfec2cf [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:27.322619 18063 raft_consensus.cc:3037] T 78eaf92538c8474da872d64a5b07c974 P 584487bc06a5401193d35b3915731633 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:27.323419 18063 raft_consensus.cc:2454] T 78eaf92538c8474da872d64a5b07c974 P 584487bc06a5401193d35b3915731633 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6496c16c497e43eab9c00ac30cbf45fc in term 1.
I20260924 07:13:27.323433 18325 raft_consensus.cc:2454] T 78eaf92538c8474da872d64a5b07c974 P c486c9c9aec64cdb8cba5fecfdfec2cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6496c16c497e43eab9c00ac30cbf45fc in term 1.
I20260924 07:13:27.323571 18127 leader_election.cc:302] T 78eaf92538c8474da872d64a5b07c974 P 6496c16c497e43eab9c00ac30cbf45fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 584487bc06a5401193d35b3915731633, 6496c16c497e43eab9c00ac30cbf45fc; no voters:
I20260924 07:13:27.323678 18393 raft_consensus.cc:2781] T 78eaf92538c8474da872d64a5b07c974 P 6496c16c497e43eab9c00ac30cbf45fc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:13:27.323827 18393 raft_consensus.cc:686] T 78eaf92538c8474da872d64a5b07c974 P 6496c16c497e43eab9c00ac30cbf45fc [term 1 LEADER]: Becoming Leader. State: Replica: 6496c16c497e43eab9c00ac30cbf45fc, State: Running, Role: LEADER
I20260924 07:13:27.323944 18393 consensus_queue.cc:237] T 78eaf92538c8474da872d64a5b07c974 P 6496c16c497e43eab9c00ac30cbf45fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486c9c9aec64cdb8cba5fecfdfec2cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45743 } } peers { permanent_uuid: "584487bc06a5401193d35b3915731633" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43827 } } peers { permanent_uuid: "6496c16c497e43eab9c00ac30cbf45fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40235 } }
I20260924 07:13:27.324831 17920 catalog_manager.cc:5495] T 78eaf92538c8474da872d64a5b07c974 P 6496c16c497e43eab9c00ac30cbf45fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 6496c16c497e43eab9c00ac30cbf45fc (127.0.98.2). New cstate: current_term: 1 leader_uuid: "6496c16c497e43eab9c00ac30cbf45fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c486c9c9aec64cdb8cba5fecfdfec2cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "584487bc06a5401193d35b3915731633" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6496c16c497e43eab9c00ac30cbf45fc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40235 } health_report { overall_health: HEALTHY } } }
I20260924 07:13:27.345286 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:27.342875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43084 (local address 127.0.98.2:40235)
0924 07:13:27.342940 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:27.342942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:27.342952 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:13:27.343024 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:27.343025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:27.343040 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:27.343113 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:13:27.343257 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:27.343719 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:27.344462 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:27.344670 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:27.344762 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:27.344925 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:13:27.345034 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:13:27.345130 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:27.345149 (+ 19us) server_negotiation.cc:299] Negotiation successful
0924 07:13:27.345169 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 07:13:27.347090 18325 raft_consensus.cc:1261] T 78eaf92538c8474da872d64a5b07c974 P c486c9c9aec64cdb8cba5fecfdfec2cf [term 1 FOLLOWER]: Refusing update from remote peer 6496c16c497e43eab9c00ac30cbf45fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:13:27.347092 18063 raft_consensus.cc:1261] T 78eaf92538c8474da872d64a5b07c974 P 584487bc06a5401193d35b3915731633 [term 1 FOLLOWER]: Refusing update from remote peer 6496c16c497e43eab9c00ac30cbf45fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:13:27.347502 18395 consensus_queue.cc:1035] T 78eaf92538c8474da872d64a5b07c974 P 6496c16c497e43eab9c00ac30cbf45fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "584487bc06a5401193d35b3915731633" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:13:27.347636 18393 consensus_queue.cc:1035] T 78eaf92538c8474da872d64a5b07c974 P 6496c16c497e43eab9c00ac30cbf45fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c486c9c9aec64cdb8cba5fecfdfec2cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:13:27.350600 18400 mvcc.cc:205] Tried to move back new op lower bound from 7332798494090940416 to 7332798493999484928. Current Snapshot: MvccSnapshot[applied={T|T < 7332798494090940416}]
I20260924 07:13:27.350734 18399 mvcc.cc:205] Tried to move back new op lower bound from 7332798494090940416 to 7332798493999484928. Current Snapshot: MvccSnapshot[applied={T|T < 7332798494090940416 or (T in {7332798494090940416})}]
I20260924 07:13:27.352057 17921 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:13:27.352154 17921 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:13:27.352901 17921 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c6c78acfca343b987880f8f85cec6fe: Sending DeleteTablet for 3 replicas of tablet 78eaf92538c8474da872d64a5b07c974
I20260924 07:13:27.353252 18043 tablet_service.cc:1514] Processing DeleteTablet for tablet 78eaf92538c8474da872d64a5b07c974 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:13:27 UTC) from {username='slave'} at 127.0.0.1:54752
I20260924 07:13:27.353246 18174 tablet_service.cc:1514] Processing DeleteTablet for tablet 78eaf92538c8474da872d64a5b07c974 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:13:27 UTC) from {username='slave'} at 127.0.0.1:43082
I20260924 07:13:27.353480 18407 tablet_replica.cc:330] T 78eaf92538c8474da872d64a5b07c974 P 584487bc06a5401193d35b3915731633: stopping tablet replica
I20260924 07:13:27.353462 18305 tablet_service.cc:1514] Processing DeleteTablet for tablet 78eaf92538c8474da872d64a5b07c974 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:13:27 UTC) from {username='slave'} at 127.0.0.1:41622
I20260924 07:13:27.353578 18407 raft_consensus.cc:2229] T 78eaf92538c8474da872d64a5b07c974 P 584487bc06a5401193d35b3915731633 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:13:27.353660 18409 tablet_replica.cc:330] T 78eaf92538c8474da872d64a5b07c974 P c486c9c9aec64cdb8cba5fecfdfec2cf: stopping tablet replica
I20260924 07:13:27.353781 18409 raft_consensus.cc:2229] T 78eaf92538c8474da872d64a5b07c974 P c486c9c9aec64cdb8cba5fecfdfec2cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:13:27.353786 18408 tablet_replica.cc:330] T 78eaf92538c8474da872d64a5b07c974 P 6496c16c497e43eab9c00ac30cbf45fc: stopping tablet replica
I20260924 07:13:27.353868 18408 raft_consensus.cc:2229] T 78eaf92538c8474da872d64a5b07c974 P 6496c16c497e43eab9c00ac30cbf45fc [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:13:27.353894 18409 raft_consensus.cc:2258] T 78eaf92538c8474da872d64a5b07c974 P c486c9c9aec64cdb8cba5fecfdfec2cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:13:27.353952 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 17981
I20260924 07:13:27.354018 18408 raft_consensus.cc:2258] T 78eaf92538c8474da872d64a5b07c974 P 6496c16c497e43eab9c00ac30cbf45fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:13:27.354321 18409 ts_tablet_manager.cc:1905] T 78eaf92538c8474da872d64a5b07c974 P c486c9c9aec64cdb8cba5fecfdfec2cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:13:27.355356 18408 ts_tablet_manager.cc:1905] T 78eaf92538c8474da872d64a5b07c974 P 6496c16c497e43eab9c00ac30cbf45fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:13:27.356288 18409 ts_tablet_manager.cc:1918] T 78eaf92538c8474da872d64a5b07c974 P c486c9c9aec64cdb8cba5fecfdfec2cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:13:27.356349 18409 log.cc:1196] T 78eaf92538c8474da872d64a5b07c974 P c486c9c9aec64cdb8cba5fecfdfec2cf: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-2/wal/wals/78eaf92538c8474da872d64a5b07c974
I20260924 07:13:27.356611 18409 ts_tablet_manager.cc:1939] T 78eaf92538c8474da872d64a5b07c974 P c486c9c9aec64cdb8cba5fecfdfec2cf: Deleting consensus metadata
I20260924 07:13:27.357007 17905 catalog_manager.cc:4841] TS c486c9c9aec64cdb8cba5fecfdfec2cf (127.0.98.3:45743): tablet 78eaf92538c8474da872d64a5b07c974 (table test-table [id=5f317c3a8f8449ce8a1d6182337818ad]) successfully deleted
I20260924 07:13:27.357337 18408 ts_tablet_manager.cc:1918] T 78eaf92538c8474da872d64a5b07c974 P 6496c16c497e43eab9c00ac30cbf45fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:13:27.357398 18408 log.cc:1196] T 78eaf92538c8474da872d64a5b07c974 P 6496c16c497e43eab9c00ac30cbf45fc: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790233772407728-392-0/minicluster-data/ts-1/wal/wals/78eaf92538c8474da872d64a5b07c974
I20260924 07:13:27.357637 18408 ts_tablet_manager.cc:1939] T 78eaf92538c8474da872d64a5b07c974 P 6496c16c497e43eab9c00ac30cbf45fc: Deleting consensus metadata
I20260924 07:13:27.357995 17908 catalog_manager.cc:4841] TS 6496c16c497e43eab9c00ac30cbf45fc (127.0.98.2:40235): tablet 78eaf92538c8474da872d64a5b07c974 (table test-table [id=5f317c3a8f8449ce8a1d6182337818ad]) successfully deleted
W20260924 07:13:27.361235 17905 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43827 (error 108)
I20260924 07:13:27.361368 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 18112
I20260924 07:13:27.365198 18382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:27.365077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43827 (local address 127.0.0.1:54756)
0924 07:13:27.365128 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:27.365171 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 07:13:27.365355 17905 catalog_manager.cc:4584] TS 584487bc06a5401193d35b3915731633 (127.0.98.1:43827): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 78eaf92538c8474da872d64a5b07c974: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43827: connect: Connection refused (error 111)
I20260924 07:13:27.365938 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 18243
I20260924 07:13:27.370594 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 17890
2026-09-24T07:13:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (810 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:13:27 dist-test-slave-14ww krb5kdc[18413](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:13:27 dist-test-slave-14ww krb5kdc[18413](info): set up 2 sockets
Sep 24 07:13:27 dist-test-slave-14ww krb5kdc[18413](info): commencing operation
krb5kdc: starting...
W20260924 07:13:28.404561 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:13:28 dist-test-slave-14ww krb5kdc[18413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:13:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:13:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:13:28.524216 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44459
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44459
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:28.603487 18426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:28.603742 18426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:28.603777 18426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:28.604943 18426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:13:28.605024 18426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:28.605048 18426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:28.605072 18426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:13:28.605099 18426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:13:28.606562 18426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:28.606842 18426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:28.607077 18426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:28.612005 18432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:28.611991 18431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:28.611991 18434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:28.612447 18426 server_base.cc:1034] running on GCE node
I20260924 07:13:28.612656 18426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:28.612969 18426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:28.614120 18426 hybrid_clock.cc:648] HybridClock initialized: now 1790234008614105 us; error 35 us; skew 500 ppm
Sep 24 07:13:28 dist-test-slave-14ww krb5kdc[18413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:28.616675 18426 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:13:28.617142 18426 webserver.cc:460] Webserver started at http://127.0.98.62:32945/ using document root <none> and password file <none>
I20260924 07:13:28.617352 18426 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:28.617403 18426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:28.617539 18426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:28.618630 18426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "1e2c10952a42491089b536909c93ced5"
format_stamp: "Formatted at 2026-09-24 07:13:28 on dist-test-slave-14ww"
server_key: "744ef577822583c94f70f8e6ff6ff952"
server_key_iv: "2aade636d37b841ef08509e74921f53c"
server_key_version: "encryptionkey@0"
I20260924 07:13:28.618980 18426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "1e2c10952a42491089b536909c93ced5"
format_stamp: "Formatted at 2026-09-24 07:13:28 on dist-test-slave-14ww"
server_key: "744ef577822583c94f70f8e6ff6ff952"
server_key_iv: "2aade636d37b841ef08509e74921f53c"
server_key_version: "encryptionkey@0"
I20260924 07:13:28.620267 18426 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:13:28.621075 18441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:28.621272 18426 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:13:28.621344 18426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "1e2c10952a42491089b536909c93ced5"
format_stamp: "Formatted at 2026-09-24 07:13:28 on dist-test-slave-14ww"
server_key: "744ef577822583c94f70f8e6ff6ff952"
server_key_iv: "2aade636d37b841ef08509e74921f53c"
server_key_version: "encryptionkey@0"
I20260924 07:13:28.621433 18426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:28.633745 18426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:28.635974 18426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:28.636124 18426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:28.640262 18426 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44459
I20260924 07:13:28.640326 18493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44459 every 8 connection(s)
I20260924 07:13:28.640647 18426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:13:28.641580 18494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:28.643846 18494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e2c10952a42491089b536909c93ced5: Bootstrap starting.
I20260924 07:13:28.644698 18494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e2c10952a42491089b536909c93ced5: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:28.645072 18494 log.cc:824] T 00000000000000000000000000000000 P 1e2c10952a42491089b536909c93ced5: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:28.645776 18494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e2c10952a42491089b536909c93ced5: No bootstrap required, opened a new log
I20260924 07:13:28.647388 18494 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e2c10952a42491089b536909c93ced5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e2c10952a42491089b536909c93ced5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44459 } }
I20260924 07:13:28.647521 18494 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e2c10952a42491089b536909c93ced5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:28.647542 18494 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e2c10952a42491089b536909c93ced5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2c10952a42491089b536909c93ced5, State: Initialized, Role: FOLLOWER
I20260924 07:13:28.647719 18494 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e2c10952a42491089b536909c93ced5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e2c10952a42491089b536909c93ced5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44459 } }
I20260924 07:13:28.647785 18494 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e2c10952a42491089b536909c93ced5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:13:28.647806 18494 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e2c10952a42491089b536909c93ced5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:13:28.647862 18494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e2c10952a42491089b536909c93ced5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:28.648409 18494 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e2c10952a42491089b536909c93ced5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e2c10952a42491089b536909c93ced5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44459 } }
I20260924 07:13:28.648521 18494 leader_election.cc:302] T 00000000000000000000000000000000 P 1e2c10952a42491089b536909c93ced5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e2c10952a42491089b536909c93ced5; no voters:
I20260924 07:13:28.648715 18494 leader_election.cc:288] T 00000000000000000000000000000000 P 1e2c10952a42491089b536909c93ced5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:13:28.648716 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 18426
I20260924 07:13:28.648761 18497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e2c10952a42491089b536909c93ced5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:13:28.648929 18497 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e2c10952a42491089b536909c93ced5 [term 1 LEADER]: Becoming Leader. State: Replica: 1e2c10952a42491089b536909c93ced5, State: Running, Role: LEADER
I20260924 07:13:28.648936 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:13:28.649092 18494 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e2c10952a42491089b536909c93ced5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:13:28.649151 392 external_mini_cluster.cc:1428] Setting key 5e64df5da80fa9e3655ad2ccd545d378
I20260924 07:13:28.649083 18497 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e2c10952a42491089b536909c93ced5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e2c10952a42491089b536909c93ced5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44459 } }
I20260924 07:13:28.649746 18498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e2c10952a42491089b536909c93ced5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e2c10952a42491089b536909c93ced5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e2c10952a42491089b536909c93ced5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44459 } } }
I20260924 07:13:28.649760 18499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e2c10952a42491089b536909c93ced5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e2c10952a42491089b536909c93ced5. Latest consensus state: current_term: 1 leader_uuid: "1e2c10952a42491089b536909c93ced5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e2c10952a42491089b536909c93ced5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44459 } } }
I20260924 07:13:28.649935 18498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e2c10952a42491089b536909c93ced5 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:28.650123 18499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e2c10952a42491089b536909c93ced5 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:28.650597 18512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:13:28.651304 18512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:13:28.652884 18512 catalog_manager.cc:1347] Generated new cluster ID: 7cf21a3055c74f03ac4f825c2a59467b
I20260924 07:13:28.652940 18512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 07:13:28 dist-test-slave-14ww krb5kdc[18413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:13:28.658068 18512 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:13:28.658726 18512 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:13:28.659834 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:28.649596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35102 (local address 127.0.98.62:44459)
0924 07:13:28.649922 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:28.650174 (+ 252us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:28.650209 (+ 35us) server_negotiation.cc:407] Connection header received
0924 07:13:28.650340 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:28.650348 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:28.650412 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:28.650566 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:28.651284 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:28.652534 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:28.653474 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:28.653907 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:28.654022 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:28.656298 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:28.656302 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:28.656302 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:28.656316 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:28.658280 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:28.658649 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:28.658652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:28.658652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:28.658743 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:28.659002 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:28.659003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:28.659003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:28.659214 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:28.659402 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:28.659467 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 07:13:28.659627 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":88,"threads_started":1}
I20260924 07:13:28.664259 18512 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e2c10952a42491089b536909c93ced5: Generated new TSK 0
I20260924 07:13:28.664431 18512 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 07:13:28.697242 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44459
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:28.779362 18520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:28.779556 18520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:28.779579 18520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:28.780725 18520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:28.780833 18520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:28.780889 18520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:13:28.782498 18520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:28.782737 18520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:28.782994 18520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:28.787873 18525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:28.787964 18528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:28.787914 18526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:28.788069 18520 server_base.cc:1034] running on GCE node
I20260924 07:13:28.788287 18520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:28.788527 18520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:28.789686 18520 hybrid_clock.cc:648] HybridClock initialized: now 1790234008789656 us; error 46 us; skew 500 ppm
Sep 24 07:13:28 dist-test-slave-14ww krb5kdc[18413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:28.792253 18520 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:13:28.792879 18520 webserver.cc:460] Webserver started at http://127.0.98.1:36455/ using document root <none> and password file <none>
I20260924 07:13:28.793150 18520 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:28.793226 18520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:28.793386 18520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:28.794416 18520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-0/data/instance:
uuid: "eb0f8dbe777f444097f3ae6f3f57fe8e"
format_stamp: "Formatted at 2026-09-24 07:13:28 on dist-test-slave-14ww"
server_key: "8f4d70dec1b8ac5702be5e379ae5555e"
server_key_iv: "c205ec4b6ab72b09b195d69376d889c6"
server_key_version: "encryptionkey@0"
I20260924 07:13:28.794787 18520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-0/wal/instance:
uuid: "eb0f8dbe777f444097f3ae6f3f57fe8e"
format_stamp: "Formatted at 2026-09-24 07:13:28 on dist-test-slave-14ww"
server_key: "8f4d70dec1b8ac5702be5e379ae5555e"
server_key_iv: "c205ec4b6ab72b09b195d69376d889c6"
server_key_version: "encryptionkey@0"
I20260924 07:13:28.796129 18520 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:13:28.797009 18535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:28.797194 18520 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:13:28.797297 18520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-0/wal
uuid: "eb0f8dbe777f444097f3ae6f3f57fe8e"
format_stamp: "Formatted at 2026-09-24 07:13:28 on dist-test-slave-14ww"
server_key: "8f4d70dec1b8ac5702be5e379ae5555e"
server_key_iv: "c205ec4b6ab72b09b195d69376d889c6"
server_key_version: "encryptionkey@0"
I20260924 07:13:28.797394 18520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:28.809094 18520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:28.810897 18520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:28.811069 18520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:28.811285 18520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:13:28.811578 18520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:13:28.811633 18520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:28.811672 18520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:13:28.811702 18520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:28.817507 18520 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46553
I20260924 07:13:28.817566 18648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46553 every 8 connection(s)
I20260924 07:13:28.817893 18520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:13:28.822098 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 18520
I20260924 07:13:28.822216 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:13:28.822410 392 external_mini_cluster.cc:1428] Setting key a5675af4eb92867d2894741db0cf7f74
Sep 24 07:13:28 dist-test-slave-14ww krb5kdc[18413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:13:28.827527 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:28.818599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34719 (local address 127.0.98.62:44459)
0924 07:13:28.818664 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:28.818666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:28.818869 (+ 203us) server_negotiation.cc:407] Connection header received
0924 07:13:28.819755 (+ 886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:28.819756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:28.819764 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:28.819816 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:28.820380 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:28.820968 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:28.822006 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:28.822335 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:28.822452 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:28.824583 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:28.824585 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:28.824585 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:28.824592 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:28.826073 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:28.826577 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:28.826578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:28.826578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:28.826631 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:28.826917 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:28.826917 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:28.826918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:28.827015 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:28.827065 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:28.827401 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 07:13:28.827467 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 07:13:28.827826 18651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:28.818586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44459 (local address 127.0.98.1:34719)
0924 07:13:28.818745 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:28.818783 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:28.819664 (+ 881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:28.819873 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:28.819876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:28.820128 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:28.820300 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:28.820304 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:28.821072 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:28.821074 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:28.821887 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:28.821889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:28.822477 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:28.822479 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:28.822617 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:28.823109 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:28.823117 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:28.824477 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:28.826157 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:28.826164 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:28.826166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:28.826495 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:28.826701 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:28.826701 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:28.826702 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:28.826849 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:28.827086 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:28.827092 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:28.827253 (+ 161us) client_negotiation.cc:769] Sending connection context
0924 07:13:28.827374 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 07:13:28.827559 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20260924 07:13:28.828351 18649 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44459
I20260924 07:13:28.828435 18649 heartbeater.cc:464] Registering TS with master...
I20260924 07:13:28.828601 18649 heartbeater.cc:511] Master 127.0.98.62:44459 requested a full tablet report, sending...
I20260924 07:13:28.829101 18458 ts_manager.cc:194] Registered new tserver with Master: eb0f8dbe777f444097f3ae6f3f57fe8e (127.0.98.1:46553)
I20260924 07:13:28.829698 18458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34719
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 07:13:28.848105 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44459
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:28.927733 18654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:28.927953 18654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:28.927999 18654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:28.929162 18654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:28.929225 18654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:28.929270 18654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:13:28.930761 18654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:28.931026 18654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:28.931255 18654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:28.936204 18660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:28.936180 18662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:28.936265 18659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:28.936349 18654 server_base.cc:1034] running on GCE node
I20260924 07:13:28.936651 18654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:28.936895 18654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:28.938060 18654 hybrid_clock.cc:648] HybridClock initialized: now 1790234008938053 us; error 40 us; skew 500 ppm
Sep 24 07:13:28 dist-test-slave-14ww krb5kdc[18413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:28.940539 18654 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:13:28.940999 18654 webserver.cc:460] Webserver started at http://127.0.98.2:39111/ using document root <none> and password file <none>
I20260924 07:13:28.941188 18654 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:28.941227 18654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:28.941313 18654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:28.942251 18654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-1/data/instance:
uuid: "da472f67fa144c378db3ec61da0978e8"
format_stamp: "Formatted at 2026-09-24 07:13:28 on dist-test-slave-14ww"
server_key: "ca8c924df8edc3d4814a56585cd5944d"
server_key_iv: "834ee31b3f6edf2d6d9faba7561b3a95"
server_key_version: "encryptionkey@0"
I20260924 07:13:28.942548 18654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-1/wal/instance:
uuid: "da472f67fa144c378db3ec61da0978e8"
format_stamp: "Formatted at 2026-09-24 07:13:28 on dist-test-slave-14ww"
server_key: "ca8c924df8edc3d4814a56585cd5944d"
server_key_iv: "834ee31b3f6edf2d6d9faba7561b3a95"
server_key_version: "encryptionkey@0"
I20260924 07:13:28.943732 18654 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:13:28.944378 18669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:28.944552 18654 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:13:28.944617 18654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-1/wal
uuid: "da472f67fa144c378db3ec61da0978e8"
format_stamp: "Formatted at 2026-09-24 07:13:28 on dist-test-slave-14ww"
server_key: "ca8c924df8edc3d4814a56585cd5944d"
server_key_iv: "834ee31b3f6edf2d6d9faba7561b3a95"
server_key_version: "encryptionkey@0"
I20260924 07:13:28.944669 18654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:28.952281 18654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:28.954500 18654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:28.954607 18654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:28.954870 18654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:13:28.955255 18654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:13:28.955300 18654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:28.955322 18654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:13:28.955334 18654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:28.961125 18654 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45907
I20260924 07:13:28.961164 18782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45907 every 8 connection(s)
I20260924 07:13:28.961481 18654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:13:28.962404 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 18654
I20260924 07:13:28.962527 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:13:28.962688 392 external_mini_cluster.cc:1428] Setting key e0a6b867d2c7e9feab607c7276ffbe67
Sep 24 07:13:28 dist-test-slave-14ww krb5kdc[18413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 07:13:28.972841 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:28.962193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60485 (local address 127.0.98.62:44459)
0924 07:13:28.962273 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:28.962276 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:28.962479 (+ 203us) server_negotiation.cc:407] Connection header received
0924 07:13:28.963641 (+ 1162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:28.963642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:28.963654 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:28.963720 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:28.964292 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:28.965042 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:28.966770 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:28.966987 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:28.967055 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:28.969580 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:28.969581 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:28.969582 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:28.969590 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:28.971444 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:28.971854 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:28.971856 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:28.971856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:28.971924 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:28.972241 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:28.972242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:28.972242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:28.972355 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:28.972416 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:28.972662 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 07:13:28.972726 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 07:13:28.973078 18785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:28.962162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44459 (local address 127.0.98.2:60485)
0924 07:13:28.962327 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:28.962368 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:28.963541 (+ 1173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:28.963772 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:28.963776 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:28.964028 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:28.964208 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:28.964212 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:28.965275 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:28.965276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:28.966661 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:28.966664 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:28.967094 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:28.967095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:28.967251 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:28.967790 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:28.967800 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:28.969481 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:28.971554 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:28.971560 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:28.971562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:28.971771 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:28.972027 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:28.972028 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:28.972028 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:28.972166 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:28.972424 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:28.972428 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:28.972547 (+ 119us) client_negotiation.cc:769] Sending connection context
0924 07:13:28.972632 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 07:13:28.972820 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":98,"threads_started":1}
I20260924 07:13:28.973528 18783 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44459
I20260924 07:13:28.973606 18783 heartbeater.cc:464] Registering TS with master...
I20260924 07:13:28.973799 18783 heartbeater.cc:511] Master 127.0.98.62:44459 requested a full tablet report, sending...
I20260924 07:13:28.974231 18458 ts_manager.cc:194] Registered new tserver with Master: da472f67fa144c378db3ec61da0978e8 (127.0.98.2:45907)
I20260924 07:13:28.974620 18458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60485
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 07:13:28.989828 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44459
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:29.071345 18788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:29.071571 18788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:29.071595 18788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:29.072682 18788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:29.072715 18788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:29.072793 18788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:13:29.074263 18788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:29.074437 18788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:29.074611 18788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:29.079545 18793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:29.079509 18796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:29.079525 18794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:29.079614 18788 server_base.cc:1034] running on GCE node
I20260924 07:13:29.079974 18788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:29.080211 18788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:29.081374 18788 hybrid_clock.cc:648] HybridClock initialized: now 1790234009081365 us; error 31 us; skew 500 ppm
Sep 24 07:13:29 dist-test-slave-14ww krb5kdc[18413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:29.083993 18788 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:13:29.084448 18788 webserver.cc:460] Webserver started at http://127.0.98.3:39315/ using document root <none> and password file <none>
I20260924 07:13:29.084640 18788 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:29.084681 18788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:29.084826 18788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:29.085793 18788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-2/data/instance:
uuid: "737aad4d7641447386c76f37bdf42ac2"
format_stamp: "Formatted at 2026-09-24 07:13:29 on dist-test-slave-14ww"
server_key: "337b5afbfc453c1ce5e7e0ffce85d405"
server_key_iv: "61a60483ef8e8e0a541511f803ecee9e"
server_key_version: "encryptionkey@0"
I20260924 07:13:29.086088 18788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-2/wal/instance:
uuid: "737aad4d7641447386c76f37bdf42ac2"
format_stamp: "Formatted at 2026-09-24 07:13:29 on dist-test-slave-14ww"
server_key: "337b5afbfc453c1ce5e7e0ffce85d405"
server_key_iv: "61a60483ef8e8e0a541511f803ecee9e"
server_key_version: "encryptionkey@0"
I20260924 07:13:29.087307 18788 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:13:29.088043 18803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:29.088285 18788 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:13:29.088383 18788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-2/wal
uuid: "737aad4d7641447386c76f37bdf42ac2"
format_stamp: "Formatted at 2026-09-24 07:13:29 on dist-test-slave-14ww"
server_key: "337b5afbfc453c1ce5e7e0ffce85d405"
server_key_iv: "61a60483ef8e8e0a541511f803ecee9e"
server_key_version: "encryptionkey@0"
I20260924 07:13:29.088464 18788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:29.100618 18788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:29.102388 18788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:29.102492 18788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:29.102645 18788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:13:29.102963 18788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:13:29.102998 18788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:29.103018 18788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:13:29.103065 18788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:29.108917 18788 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46707
I20260924 07:13:29.108961 18916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46707 every 8 connection(s)
I20260924 07:13:29.109247 18788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:13:29.114228 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 18788
I20260924 07:13:29.114341 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:13:29.114523 392 external_mini_cluster.cc:1428] Setting key 195170d1d66f1636cfcdcad5e4affe2f
Sep 24 07:13:29 dist-test-slave-14ww krb5kdc[18413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234009, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:13:29.118863 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:29.109874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37807 (local address 127.0.98.62:44459)
0924 07:13:29.109970 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:29.109973 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:29.110139 (+ 166us) server_negotiation.cc:407] Connection header received
0924 07:13:29.111060 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:29.111061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:29.111074 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:29.111127 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:29.111711 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.112222 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.113317 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.113523 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.113607 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:29.116184 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:29.116185 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:29.116185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:29.116192 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:29.117505 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:29.117910 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:29.117912 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:29.117912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:29.117971 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:29.118238 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:29.118239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:29.118239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:29.118353 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:29.118431 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:29.118689 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 07:13:29.118775 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 07:13:29.119041 18919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:29.109847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44459 (local address 127.0.98.3:37807)
0924 07:13:29.110025 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:29.110061 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:29.110953 (+ 892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:29.111197 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:29.111200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:29.111445 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:29.111617 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:29.111621 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.112315 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.112316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:29.113220 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:29.113223 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.113616 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.113617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:29.113738 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:29.114262 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:29.114270 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:29.116079 (+ 1809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:29.117597 (+ 1518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:29.117603 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:29.117605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:29.117817 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:29.118048 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:29.118049 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:29.118049 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:29.118169 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:29.118422 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:29.118429 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:29.118563 (+ 134us) client_negotiation.cc:769] Sending connection context
0924 07:13:29.118668 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 07:13:29.118818 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":80,"threads_started":1}
I20260924 07:13:29.119477 18917 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44459
I20260924 07:13:29.119552 18917 heartbeater.cc:464] Registering TS with master...
I20260924 07:13:29.119678 18917 heartbeater.cc:511] Master 127.0.98.62:44459 requested a full tablet report, sending...
I20260924 07:13:29.119989 18458 ts_manager.cc:194] Registered new tserver with Master: 737aad4d7641447386c76f37bdf42ac2 (127.0.98.3:46707)
I20260924 07:13:29.120373 18458 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37807
I20260924 07:13:29.126283 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:13:29.133646 18505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:29.127623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35106 (local address 127.0.98.62:44459)
0924 07:13:29.127687 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:29.127689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:29.127788 (+ 99us) server_negotiation.cc:407] Connection header received
0924 07:13:29.127824 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:29.127825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:29.127833 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:29.127878 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:29.128409 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.128964 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.129686 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.129858 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.129906 (+ 48us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:29.131120 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:29.131121 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:29.131122 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:29.131127 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:29.132511 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:29.132946 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:29.132947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:29.132947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:29.132997 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:29.133170 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:29.133170 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:29.133171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:29.133280 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:29.133351 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:29.133461 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 07:13:29.133518 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 07:13:29.135179 18458 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:13:29.136790 18458 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:13:29.142771 18925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:29.138798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46707 (local address 127.0.0.1:50714)
0924 07:13:29.139047 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:29.139087 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:29.139155 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:29.139378 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:29.139382 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:29.139385 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:29.139460 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:29.139462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.140543 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.140544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:29.141789 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:29.141791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.142469 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.142470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:29.142547 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:29.142561 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 07:13:29.142654 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 07:13:29.142702 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":52,"threads_started":1}
I20260924 07:13:29.142786 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:29.138860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50714 (local address 127.0.98.3:46707)
0924 07:13:29.139211 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:29.139215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:29.139224 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:13:29.139257 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:29.139259 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:29.139280 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:29.139371 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:29.139713 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.140419 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.141883 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.142368 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.142465 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:29.142529 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:29.142668 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 07:13:29.142707 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
I20260924 07:13:29.143003 18926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:29.138832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46553 (local address 127.0.0.1:50874)
0924 07:13:29.139354 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:29.139362 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:29.139407 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:29.139608 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:29.139609 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:29.139611 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:29.139649 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:29.139650 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.141004 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.141005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:29.141843 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:29.141845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.142783 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.142784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:29.142843 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:29.142847 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:13:29.142892 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 07:13:29.142928 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":21,"thread_start_us":71,"threads_started":1}
I20260924 07:13:29.143353 18929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:29.139044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50874 (local address 127.0.98.1:46553)
0924 07:13:29.139191 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:29.139196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:29.139467 (+ 271us) server_negotiation.cc:407] Connection header received
0924 07:13:29.139510 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:29.139513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:29.139529 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:29.139600 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:29.139768 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.140873 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.141918 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.142640 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.142700 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:29.142750 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:29.143234 (+ 484us) server_negotiation.cc:299] Negotiation successful
0924 07:13:29.143279 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
I20260924 07:13:29.143714 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:29.138831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45907 (local address 127.0.0.1:40676)
0924 07:13:29.139237 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:29.139248 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:29.139288 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:29.139539 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:29.139540 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:29.139542 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:29.139592 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:29.139593 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.140987 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.140988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:29.142428 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:29.142430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.143485 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.143486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:29.143551 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:29.143554 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:13:29.143615 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 07:13:29.143653 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":25,"thread_start_us":63,"threads_started":1}
I20260924 07:13:29.143577 18851 tablet_service.cc:1467] Processing CreateTablet for tablet 911eecfd5bff48fcb28f97e1da980f54 (DEFAULT_TABLE table=test-table [id=eb3e72bb50824206a23927bf7033fb74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:13:29.143972 18851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 911eecfd5bff48fcb28f97e1da980f54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:29.144153 18930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:29.139071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40676 (local address 127.0.98.2:45907)
0924 07:13:29.139287 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:29.139292 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:29.139302 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:13:29.139350 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:29.139352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:29.139370 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:29.139440 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:29.139671 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.140892 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.142525 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.143395 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.143476 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:29.143529 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:29.144034 (+ 505us) server_negotiation.cc:299] Negotiation successful
0924 07:13:29.144078 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":76,"threads_started":1}
I20260924 07:13:29.144207 18583 tablet_service.cc:1467] Processing CreateTablet for tablet 911eecfd5bff48fcb28f97e1da980f54 (DEFAULT_TABLE table=test-table [id=eb3e72bb50824206a23927bf7033fb74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:13:29.144479 18583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 911eecfd5bff48fcb28f97e1da980f54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:29.144794 18717 tablet_service.cc:1467] Processing CreateTablet for tablet 911eecfd5bff48fcb28f97e1da980f54 (DEFAULT_TABLE table=test-table [id=eb3e72bb50824206a23927bf7033fb74]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:13:29.145089 18717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 911eecfd5bff48fcb28f97e1da980f54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:29.146694 18931 tablet_bootstrap.cc:492] T 911eecfd5bff48fcb28f97e1da980f54 P 737aad4d7641447386c76f37bdf42ac2: Bootstrap starting.
I20260924 07:13:29.147347 18931 tablet_bootstrap.cc:654] T 911eecfd5bff48fcb28f97e1da980f54 P 737aad4d7641447386c76f37bdf42ac2: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:29.147754 18932 tablet_bootstrap.cc:492] T 911eecfd5bff48fcb28f97e1da980f54 P eb0f8dbe777f444097f3ae6f3f57fe8e: Bootstrap starting.
I20260924 07:13:29.148015 18931 log.cc:824] T 911eecfd5bff48fcb28f97e1da980f54 P 737aad4d7641447386c76f37bdf42ac2: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:29.148466 18932 tablet_bootstrap.cc:654] T 911eecfd5bff48fcb28f97e1da980f54 P eb0f8dbe777f444097f3ae6f3f57fe8e: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:29.148630 18934 tablet_bootstrap.cc:492] T 911eecfd5bff48fcb28f97e1da980f54 P da472f67fa144c378db3ec61da0978e8: Bootstrap starting.
I20260924 07:13:29.149107 18932 log.cc:824] T 911eecfd5bff48fcb28f97e1da980f54 P eb0f8dbe777f444097f3ae6f3f57fe8e: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:29.149295 18931 tablet_bootstrap.cc:492] T 911eecfd5bff48fcb28f97e1da980f54 P 737aad4d7641447386c76f37bdf42ac2: No bootstrap required, opened a new log
I20260924 07:13:29.149298 18934 tablet_bootstrap.cc:654] T 911eecfd5bff48fcb28f97e1da980f54 P da472f67fa144c378db3ec61da0978e8: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:29.149403 18931 ts_tablet_manager.cc:1397] T 911eecfd5bff48fcb28f97e1da980f54 P 737aad4d7641447386c76f37bdf42ac2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 07:13:29.149915 18932 tablet_bootstrap.cc:492] T 911eecfd5bff48fcb28f97e1da980f54 P eb0f8dbe777f444097f3ae6f3f57fe8e: No bootstrap required, opened a new log
I20260924 07:13:29.149979 18932 ts_tablet_manager.cc:1397] T 911eecfd5bff48fcb28f97e1da980f54 P eb0f8dbe777f444097f3ae6f3f57fe8e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:13:29.149958 18934 log.cc:824] T 911eecfd5bff48fcb28f97e1da980f54 P da472f67fa144c378db3ec61da0978e8: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:29.150662 18934 tablet_bootstrap.cc:492] T 911eecfd5bff48fcb28f97e1da980f54 P da472f67fa144c378db3ec61da0978e8: No bootstrap required, opened a new log
I20260924 07:13:29.150784 18934 ts_tablet_manager.cc:1397] T 911eecfd5bff48fcb28f97e1da980f54 P da472f67fa144c378db3ec61da0978e8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:13:29.151379 18931 raft_consensus.cc:348] T 911eecfd5bff48fcb28f97e1da980f54 P 737aad4d7641447386c76f37bdf42ac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737aad4d7641447386c76f37bdf42ac2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46707 } } peers { permanent_uuid: "da472f67fa144c378db3ec61da0978e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45907 } } peers { permanent_uuid: "eb0f8dbe777f444097f3ae6f3f57fe8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46553 } }
I20260924 07:13:29.151561 18931 raft_consensus.cc:374] T 911eecfd5bff48fcb28f97e1da980f54 P 737aad4d7641447386c76f37bdf42ac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:29.151629 18931 raft_consensus.cc:729] T 911eecfd5bff48fcb28f97e1da980f54 P 737aad4d7641447386c76f37bdf42ac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 737aad4d7641447386c76f37bdf42ac2, State: Initialized, Role: FOLLOWER
I20260924 07:13:29.151757 18931 consensus_queue.cc:260] T 911eecfd5bff48fcb28f97e1da980f54 P 737aad4d7641447386c76f37bdf42ac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737aad4d7641447386c76f37bdf42ac2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46707 } } peers { permanent_uuid: "da472f67fa144c378db3ec61da0978e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45907 } } peers { permanent_uuid: "eb0f8dbe777f444097f3ae6f3f57fe8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46553 } }
I20260924 07:13:29.152031 18931 ts_tablet_manager.cc:1428] T 911eecfd5bff48fcb28f97e1da980f54 P 737aad4d7641447386c76f37bdf42ac2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 07:13:29.151944 18932 raft_consensus.cc:348] T 911eecfd5bff48fcb28f97e1da980f54 P eb0f8dbe777f444097f3ae6f3f57fe8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737aad4d7641447386c76f37bdf42ac2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46707 } } peers { permanent_uuid: "da472f67fa144c378db3ec61da0978e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45907 } } peers { permanent_uuid: "eb0f8dbe777f444097f3ae6f3f57fe8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46553 } }
I20260924 07:13:29.152097 18932 raft_consensus.cc:374] T 911eecfd5bff48fcb28f97e1da980f54 P eb0f8dbe777f444097f3ae6f3f57fe8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:29.152150 18917 heartbeater.cc:503] Master 127.0.98.62:44459 was elected leader, sending a full tablet report...
I20260924 07:13:29.152179 18932 raft_consensus.cc:729] T 911eecfd5bff48fcb28f97e1da980f54 P eb0f8dbe777f444097f3ae6f3f57fe8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb0f8dbe777f444097f3ae6f3f57fe8e, State: Initialized, Role: FOLLOWER
I20260924 07:13:29.152302 18932 consensus_queue.cc:260] T 911eecfd5bff48fcb28f97e1da980f54 P eb0f8dbe777f444097f3ae6f3f57fe8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737aad4d7641447386c76f37bdf42ac2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46707 } } peers { permanent_uuid: "da472f67fa144c378db3ec61da0978e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45907 } } peers { permanent_uuid: "eb0f8dbe777f444097f3ae6f3f57fe8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46553 } }
I20260924 07:13:29.152662 18932 ts_tablet_manager.cc:1428] T 911eecfd5bff48fcb28f97e1da980f54 P eb0f8dbe777f444097f3ae6f3f57fe8e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 07:13:29.152666 18649 heartbeater.cc:503] Master 127.0.98.62:44459 was elected leader, sending a full tablet report...
I20260924 07:13:29.152755 18934 raft_consensus.cc:348] T 911eecfd5bff48fcb28f97e1da980f54 P da472f67fa144c378db3ec61da0978e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737aad4d7641447386c76f37bdf42ac2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46707 } } peers { permanent_uuid: "da472f67fa144c378db3ec61da0978e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45907 } } peers { permanent_uuid: "eb0f8dbe777f444097f3ae6f3f57fe8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46553 } }
I20260924 07:13:29.152909 18934 raft_consensus.cc:374] T 911eecfd5bff48fcb28f97e1da980f54 P da472f67fa144c378db3ec61da0978e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:29.152941 18934 raft_consensus.cc:729] T 911eecfd5bff48fcb28f97e1da980f54 P da472f67fa144c378db3ec61da0978e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da472f67fa144c378db3ec61da0978e8, State: Initialized, Role: FOLLOWER
I20260924 07:13:29.153045 18934 consensus_queue.cc:260] T 911eecfd5bff48fcb28f97e1da980f54 P da472f67fa144c378db3ec61da0978e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737aad4d7641447386c76f37bdf42ac2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46707 } } peers { permanent_uuid: "da472f67fa144c378db3ec61da0978e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45907 } } peers { permanent_uuid: "eb0f8dbe777f444097f3ae6f3f57fe8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46553 } }
I20260924 07:13:29.153298 18934 ts_tablet_manager.cc:1428] T 911eecfd5bff48fcb28f97e1da980f54 P da472f67fa144c378db3ec61da0978e8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:13:29.153405 18783 heartbeater.cc:503] Master 127.0.98.62:44459 was elected leader, sending a full tablet report...
W20260924 07:13:29.212049 18784 tablet.cc:2369] T 911eecfd5bff48fcb28f97e1da980f54 P da472f67fa144c378db3ec61da0978e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:13:29.226341 18937 raft_consensus.cc:482] T 911eecfd5bff48fcb28f97e1da980f54 P 737aad4d7641447386c76f37bdf42ac2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:13:29.226444 18937 raft_consensus.cc:504] T 911eecfd5bff48fcb28f97e1da980f54 P 737aad4d7641447386c76f37bdf42ac2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737aad4d7641447386c76f37bdf42ac2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46707 } } peers { permanent_uuid: "da472f67fa144c378db3ec61da0978e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45907 } } peers { permanent_uuid: "eb0f8dbe777f444097f3ae6f3f57fe8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46553 } }
I20260924 07:13:29.226876 18937 leader_election.cc:288] T 911eecfd5bff48fcb28f97e1da980f54 P 737aad4d7641447386c76f37bdf42ac2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da472f67fa144c378db3ec61da0978e8 (127.0.98.2:45907), eb0f8dbe777f444097f3ae6f3f57fe8e (127.0.98.1:46553)
I20260924 07:13:29.230821 18930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:29.227127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36111 (local address 127.0.98.2:45907)
0924 07:13:29.227190 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:29.227197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:29.227208 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:13:29.227235 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:29.227236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:29.227248 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:29.227308 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:29.227505 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.228291 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.229870 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.230457 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.230527 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:29.230538 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:29.230711 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 07:13:29.230753 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 07:13:29.230834 18919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:29.227039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45907 (local address 127.0.98.3:36111)
0924 07:13:29.227096 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:29.227108 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:29.227157 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:29.227317 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:29.227318 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:29.227320 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:29.227410 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:29.227419 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.228402 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.228403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:29.229748 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:29.229752 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.230541 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.230543 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:29.230623 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:29.230626 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:13:29.230686 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 07:13:29.230740 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":85}
I20260924 07:13:29.230882 18929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:29.227200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60313 (local address 127.0.98.1:46553)
0924 07:13:29.227254 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:29.227256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:29.227342 (+ 86us) server_negotiation.cc:407] Connection header received
0924 07:13:29.227375 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:29.227376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:29.227389 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:29.227447 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:29.227603 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.228290 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.229870 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.230446 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.230527 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:29.230538 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:29.230710 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 07:13:29.230756 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 07:13:29.230846 18940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:29.227096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46553 (local address 127.0.98.3:60313)
0924 07:13:29.227269 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:29.227278 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:29.227326 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:29.227465 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:29.227466 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:29.227467 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:29.227526 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:29.227528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.228401 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.228403 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:29.229748 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:29.229752 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.230542 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.230542 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:29.230623 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:29.230626 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:13:29.230686 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 07:13:29.230740 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":51,"thread_start_us":106,"threads_started":1}
I20260924 07:13:29.231177 18737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "911eecfd5bff48fcb28f97e1da980f54" candidate_uuid: "737aad4d7641447386c76f37bdf42ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da472f67fa144c378db3ec61da0978e8" is_pre_election: true
I20260924 07:13:29.231318 18737 raft_consensus.cc:2454] T 911eecfd5bff48fcb28f97e1da980f54 P da472f67fa144c378db3ec61da0978e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 737aad4d7641447386c76f37bdf42ac2 in term 0.
I20260924 07:13:29.231557 18805 leader_election.cc:302] T 911eecfd5bff48fcb28f97e1da980f54 P 737aad4d7641447386c76f37bdf42ac2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 737aad4d7641447386c76f37bdf42ac2, da472f67fa144c378db3ec61da0978e8; no voters:
I20260924 07:13:29.231689 18937 raft_consensus.cc:2781] T 911eecfd5bff48fcb28f97e1da980f54 P 737aad4d7641447386c76f37bdf42ac2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:13:29.231660 18603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "911eecfd5bff48fcb28f97e1da980f54" candidate_uuid: "737aad4d7641447386c76f37bdf42ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb0f8dbe777f444097f3ae6f3f57fe8e" is_pre_election: true
I20260924 07:13:29.231752 18937 raft_consensus.cc:482] T 911eecfd5bff48fcb28f97e1da980f54 P 737aad4d7641447386c76f37bdf42ac2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:13:29.231801 18937 raft_consensus.cc:3037] T 911eecfd5bff48fcb28f97e1da980f54 P 737aad4d7641447386c76f37bdf42ac2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:29.231801 18603 raft_consensus.cc:2454] T 911eecfd5bff48fcb28f97e1da980f54 P eb0f8dbe777f444097f3ae6f3f57fe8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 737aad4d7641447386c76f37bdf42ac2 in term 0.
I20260924 07:13:29.232496 18937 raft_consensus.cc:504] T 911eecfd5bff48fcb28f97e1da980f54 P 737aad4d7641447386c76f37bdf42ac2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737aad4d7641447386c76f37bdf42ac2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46707 } } peers { permanent_uuid: "da472f67fa144c378db3ec61da0978e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45907 } } peers { permanent_uuid: "eb0f8dbe777f444097f3ae6f3f57fe8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46553 } }
I20260924 07:13:29.232630 18937 leader_election.cc:288] T 911eecfd5bff48fcb28f97e1da980f54 P 737aad4d7641447386c76f37bdf42ac2 [CANDIDATE]: Term 1 election: Requested vote from peers da472f67fa144c378db3ec61da0978e8 (127.0.98.2:45907), eb0f8dbe777f444097f3ae6f3f57fe8e (127.0.98.1:46553)
I20260924 07:13:29.232826 18603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "911eecfd5bff48fcb28f97e1da980f54" candidate_uuid: "737aad4d7641447386c76f37bdf42ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb0f8dbe777f444097f3ae6f3f57fe8e"
I20260924 07:13:29.232827 18737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "911eecfd5bff48fcb28f97e1da980f54" candidate_uuid: "737aad4d7641447386c76f37bdf42ac2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da472f67fa144c378db3ec61da0978e8"
I20260924 07:13:29.232918 18737 raft_consensus.cc:3037] T 911eecfd5bff48fcb28f97e1da980f54 P da472f67fa144c378db3ec61da0978e8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:29.232918 18603 raft_consensus.cc:3037] T 911eecfd5bff48fcb28f97e1da980f54 P eb0f8dbe777f444097f3ae6f3f57fe8e [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:29.233788 18603 raft_consensus.cc:2454] T 911eecfd5bff48fcb28f97e1da980f54 P eb0f8dbe777f444097f3ae6f3f57fe8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 737aad4d7641447386c76f37bdf42ac2 in term 1.
I20260924 07:13:29.233803 18737 raft_consensus.cc:2454] T 911eecfd5bff48fcb28f97e1da980f54 P da472f67fa144c378db3ec61da0978e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 737aad4d7641447386c76f37bdf42ac2 in term 1.
I20260924 07:13:29.234016 18804 leader_election.cc:302] T 911eecfd5bff48fcb28f97e1da980f54 P 737aad4d7641447386c76f37bdf42ac2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 737aad4d7641447386c76f37bdf42ac2, eb0f8dbe777f444097f3ae6f3f57fe8e; no voters:
I20260924 07:13:29.234109 18937 raft_consensus.cc:2781] T 911eecfd5bff48fcb28f97e1da980f54 P 737aad4d7641447386c76f37bdf42ac2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:13:29.234251 18937 raft_consensus.cc:686] T 911eecfd5bff48fcb28f97e1da980f54 P 737aad4d7641447386c76f37bdf42ac2 [term 1 LEADER]: Becoming Leader. State: Replica: 737aad4d7641447386c76f37bdf42ac2, State: Running, Role: LEADER
I20260924 07:13:29.234388 18937 consensus_queue.cc:237] T 911eecfd5bff48fcb28f97e1da980f54 P 737aad4d7641447386c76f37bdf42ac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737aad4d7641447386c76f37bdf42ac2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46707 } } peers { permanent_uuid: "da472f67fa144c378db3ec61da0978e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45907 } } peers { permanent_uuid: "eb0f8dbe777f444097f3ae6f3f57fe8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46553 } }
I20260924 07:13:29.235172 18458 catalog_manager.cc:5495] T 911eecfd5bff48fcb28f97e1da980f54 P 737aad4d7641447386c76f37bdf42ac2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 737aad4d7641447386c76f37bdf42ac2 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "737aad4d7641447386c76f37bdf42ac2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "737aad4d7641447386c76f37bdf42ac2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da472f67fa144c378db3ec61da0978e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb0f8dbe777f444097f3ae6f3f57fe8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46553 } health_report { overall_health: UNKNOWN } } }
I20260924 07:13:29.258757 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:29.256398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50730 (local address 127.0.98.3:46707)
0924 07:13:29.256456 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:29.256458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:29.256466 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:13:29.256502 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:29.256503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:29.256517 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:29.256574 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:13:29.256682 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.257208 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.257945 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.258138 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.258191 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:29.258355 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:13:29.258445 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:13:29.258520 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:29.258632 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 07:13:29.258641 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 07:13:29.260608 18737 raft_consensus.cc:1261] T 911eecfd5bff48fcb28f97e1da980f54 P da472f67fa144c378db3ec61da0978e8 [term 1 FOLLOWER]: Refusing update from remote peer 737aad4d7641447386c76f37bdf42ac2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:13:29.260601 18603 raft_consensus.cc:1261] T 911eecfd5bff48fcb28f97e1da980f54 P eb0f8dbe777f444097f3ae6f3f57fe8e [term 1 FOLLOWER]: Refusing update from remote peer 737aad4d7641447386c76f37bdf42ac2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:13:29.260991 18941 consensus_queue.cc:1035] T 911eecfd5bff48fcb28f97e1da980f54 P 737aad4d7641447386c76f37bdf42ac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb0f8dbe777f444097f3ae6f3f57fe8e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:13:29.261121 18937 consensus_queue.cc:1035] T 911eecfd5bff48fcb28f97e1da980f54 P 737aad4d7641447386c76f37bdf42ac2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da472f67fa144c378db3ec61da0978e8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:13:29.263770 18945 mvcc.cc:205] Tried to move back new op lower bound from 7332798501928222720 to 7332798501824724992. Current Snapshot: MvccSnapshot[applied={T|T < 7332798501928222720}]
I20260924 07:13:29.265936 18458 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:13:29.266058 18458 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:13:29.266824 18458 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e2c10952a42491089b536909c93ced5: Sending DeleteTablet for 3 replicas of tablet 911eecfd5bff48fcb28f97e1da980f54
I20260924 07:13:29.267153 18717 tablet_service.cc:1514] Processing DeleteTablet for tablet 911eecfd5bff48fcb28f97e1da980f54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:13:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40676
I20260924 07:13:29.267184 18851 tablet_service.cc:1514] Processing DeleteTablet for tablet 911eecfd5bff48fcb28f97e1da980f54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:13:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50714
I20260924 07:13:29.267168 18583 tablet_service.cc:1514] Processing DeleteTablet for tablet 911eecfd5bff48fcb28f97e1da980f54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:13:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50874
I20260924 07:13:29.267390 18954 tablet_replica.cc:330] T 911eecfd5bff48fcb28f97e1da980f54 P eb0f8dbe777f444097f3ae6f3f57fe8e: stopping tablet replica
I20260924 07:13:29.267517 18954 raft_consensus.cc:2229] T 911eecfd5bff48fcb28f97e1da980f54 P eb0f8dbe777f444097f3ae6f3f57fe8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:13:29.267547 18955 tablet_replica.cc:330] T 911eecfd5bff48fcb28f97e1da980f54 P da472f67fa144c378db3ec61da0978e8: stopping tablet replica
I20260924 07:13:29.267650 18954 raft_consensus.cc:2258] T 911eecfd5bff48fcb28f97e1da980f54 P eb0f8dbe777f444097f3ae6f3f57fe8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:13:29.267700 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 18520
I20260924 07:13:29.267704 18955 raft_consensus.cc:2229] T 911eecfd5bff48fcb28f97e1da980f54 P da472f67fa144c378db3ec61da0978e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:13:29.268285 18955 raft_consensus.cc:2258] T 911eecfd5bff48fcb28f97e1da980f54 P da472f67fa144c378db3ec61da0978e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:13:29.268998 18955 ts_tablet_manager.cc:1905] T 911eecfd5bff48fcb28f97e1da980f54 P da472f67fa144c378db3ec61da0978e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:13:29.270917 18955 ts_tablet_manager.cc:1918] T 911eecfd5bff48fcb28f97e1da980f54 P da472f67fa144c378db3ec61da0978e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:13:29.270991 18955 log.cc:1196] T 911eecfd5bff48fcb28f97e1da980f54 P da472f67fa144c378db3ec61da0978e8: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-1/wal/wals/911eecfd5bff48fcb28f97e1da980f54
I20260924 07:13:29.271255 18955 ts_tablet_manager.cc:1939] T 911eecfd5bff48fcb28f97e1da980f54 P da472f67fa144c378db3ec61da0978e8: Deleting consensus metadata
I20260924 07:13:29.271641 18443 catalog_manager.cc:4841] TS da472f67fa144c378db3ec61da0978e8 (127.0.98.2:45907): tablet 911eecfd5bff48fcb28f97e1da980f54 (table test-table [id=eb3e72bb50824206a23927bf7033fb74]) successfully deleted
W20260924 07:13:29.273909 18442 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46553 (error 108)
I20260924 07:13:29.274238 18956 tablet_replica.cc:330] T 911eecfd5bff48fcb28f97e1da980f54 P 737aad4d7641447386c76f37bdf42ac2: stopping tablet replica
I20260924 07:13:29.274355 18956 raft_consensus.cc:2229] T 911eecfd5bff48fcb28f97e1da980f54 P 737aad4d7641447386c76f37bdf42ac2 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:13:29.274463 18956 raft_consensus.cc:2258] T 911eecfd5bff48fcb28f97e1da980f54 P 737aad4d7641447386c76f37bdf42ac2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 07:13:29.274549 18927 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:13:29.274275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46553 (local address 127.0.0.1:50888)
0924 07:13:29.274324 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:29.274336 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:29.274373 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:29.274506 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46553: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 07:13:29.274664 18442 catalog_manager.cc:4584] TS eb0f8dbe777f444097f3ae6f3f57fe8e (127.0.98.1:46553): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 911eecfd5bff48fcb28f97e1da980f54: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46553: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 07:13:29.274935 18956 ts_tablet_manager.cc:1905] T 911eecfd5bff48fcb28f97e1da980f54 P 737aad4d7641447386c76f37bdf42ac2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:13:29.274999 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 18654
I20260924 07:13:29.276978 18956 ts_tablet_manager.cc:1918] T 911eecfd5bff48fcb28f97e1da980f54 P 737aad4d7641447386c76f37bdf42ac2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:13:29.277078 18956 log.cc:1196] T 911eecfd5bff48fcb28f97e1da980f54 P 737aad4d7641447386c76f37bdf42ac2: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790233772407728-392-0/minicluster-data/ts-2/wal/wals/911eecfd5bff48fcb28f97e1da980f54
I20260924 07:13:29.277374 18956 ts_tablet_manager.cc:1939] T 911eecfd5bff48fcb28f97e1da980f54 P 737aad4d7641447386c76f37bdf42ac2: Deleting consensus metadata
I20260924 07:13:29.277812 18445 catalog_manager.cc:4841] TS 737aad4d7641447386c76f37bdf42ac2 (127.0.98.3:46707): tablet 911eecfd5bff48fcb28f97e1da980f54 (table test-table [id=eb3e72bb50824206a23927bf7033fb74]) successfully deleted
I20260924 07:13:29.280439 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 18788
I20260924 07:13:29.286245 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 18426
2026-09-24T07:13:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1914 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T07:13:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:13:29Z Disabled control of system clock
I20260924 07:13:29.313015 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32975
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 07:13:29.391139 18961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:29.391351 18961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:29.391371 18961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:29.392417 18961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:13:29.392450 18961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:29.392463 18961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:29.392475 18961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:13:29.392488 18961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:13:29.393736 18961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32975
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:29.393920 18961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:29.394148 18961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:29.398892 18966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:29.398892 18967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:29.399005 18961 server_base.cc:1034] running on GCE node
W20260924 07:13:29.398955 18969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:29.399333 18961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:29.399569 18961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:29.400703 18961 hybrid_clock.cc:648] HybridClock initialized: now 1790234009400693 us; error 30 us; skew 500 ppm
I20260924 07:13:29.401847 18961 webserver.cc:460] Webserver started at http://127.0.98.62:37665/ using document root <none> and password file <none>
I20260924 07:13:29.402041 18961 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:29.402081 18961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:29.402169 18961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:29.403118 18961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "7aee4cd6b02f4ae29cbc18073cda7345"
format_stamp: "Formatted at 2026-09-24 07:13:29 on dist-test-slave-14ww"
server_key: "d8e84fe09b6a756b130732b12f45c269"
server_key_iv: "e0d86d1f746558fe9164a4107231f411"
server_key_version: "encryptionkey@0"
I20260924 07:13:29.403401 18961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "7aee4cd6b02f4ae29cbc18073cda7345"
format_stamp: "Formatted at 2026-09-24 07:13:29 on dist-test-slave-14ww"
server_key: "d8e84fe09b6a756b130732b12f45c269"
server_key_iv: "e0d86d1f746558fe9164a4107231f411"
server_key_version: "encryptionkey@0"
I20260924 07:13:29.404609 18961 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:13:29.405366 18975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:29.405536 18961 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:29.405622 18961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "7aee4cd6b02f4ae29cbc18073cda7345"
format_stamp: "Formatted at 2026-09-24 07:13:29 on dist-test-slave-14ww"
server_key: "d8e84fe09b6a756b130732b12f45c269"
server_key_iv: "e0d86d1f746558fe9164a4107231f411"
server_key_version: "encryptionkey@0"
I20260924 07:13:29.405690 18961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:29.417655 18961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:29.417920 18961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:29.418058 18961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:29.421940 18961 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32975
I20260924 07:13:29.421993 19027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32975 every 8 connection(s)
I20260924 07:13:29.422307 18961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:13:29.423215 19028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:29.425446 19028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aee4cd6b02f4ae29cbc18073cda7345: Bootstrap starting.
I20260924 07:13:29.426009 19028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7aee4cd6b02f4ae29cbc18073cda7345: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:29.426304 19028 log.cc:824] T 00000000000000000000000000000000 P 7aee4cd6b02f4ae29cbc18073cda7345: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:29.426834 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 18961
I20260924 07:13:29.426951 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:13:29.427052 19028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aee4cd6b02f4ae29cbc18073cda7345: No bootstrap required, opened a new log
I20260924 07:13:29.427169 392 external_mini_cluster.cc:1428] Setting key f2c265cab1405f41392d189b056fe843
I20260924 07:13:29.429962 19028 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7aee4cd6b02f4ae29cbc18073cda7345 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aee4cd6b02f4ae29cbc18073cda7345" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32975 } }
I20260924 07:13:29.430122 19028 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7aee4cd6b02f4ae29cbc18073cda7345 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:29.430151 19028 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7aee4cd6b02f4ae29cbc18073cda7345 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aee4cd6b02f4ae29cbc18073cda7345, State: Initialized, Role: FOLLOWER
I20260924 07:13:29.430254 19028 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7aee4cd6b02f4ae29cbc18073cda7345 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aee4cd6b02f4ae29cbc18073cda7345" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32975 } }
I20260924 07:13:29.430317 19028 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7aee4cd6b02f4ae29cbc18073cda7345 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:13:29.430346 19028 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7aee4cd6b02f4ae29cbc18073cda7345 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:13:29.430392 19028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7aee4cd6b02f4ae29cbc18073cda7345 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:29.430922 19028 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7aee4cd6b02f4ae29cbc18073cda7345 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aee4cd6b02f4ae29cbc18073cda7345" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32975 } }
I20260924 07:13:29.431048 19028 leader_election.cc:302] T 00000000000000000000000000000000 P 7aee4cd6b02f4ae29cbc18073cda7345 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7aee4cd6b02f4ae29cbc18073cda7345; no voters:
I20260924 07:13:29.431205 19028 leader_election.cc:288] T 00000000000000000000000000000000 P 7aee4cd6b02f4ae29cbc18073cda7345 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:13:29.431236 19033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7aee4cd6b02f4ae29cbc18073cda7345 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:13:29.431332 19033 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7aee4cd6b02f4ae29cbc18073cda7345 [term 1 LEADER]: Becoming Leader. State: Replica: 7aee4cd6b02f4ae29cbc18073cda7345, State: Running, Role: LEADER
I20260924 07:13:29.431453 19028 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7aee4cd6b02f4ae29cbc18073cda7345 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:13:29.431470 19033 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7aee4cd6b02f4ae29cbc18073cda7345 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aee4cd6b02f4ae29cbc18073cda7345" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32975 } }
I20260924 07:13:29.431778 19033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aee4cd6b02f4ae29cbc18073cda7345 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7aee4cd6b02f4ae29cbc18073cda7345. Latest consensus state: current_term: 1 leader_uuid: "7aee4cd6b02f4ae29cbc18073cda7345" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aee4cd6b02f4ae29cbc18073cda7345" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32975 } } }
I20260924 07:13:29.431874 19033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aee4cd6b02f4ae29cbc18073cda7345 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:29.431792 19034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aee4cd6b02f4ae29cbc18073cda7345 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7aee4cd6b02f4ae29cbc18073cda7345" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aee4cd6b02f4ae29cbc18073cda7345" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32975 } } }
I20260924 07:13:29.432061 19034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aee4cd6b02f4ae29cbc18073cda7345 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:29.432931 19039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:13:29.433420 19032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:29.427670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57458 (local address 127.0.98.62:32975)
0924 07:13:29.427906 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:29.427915 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:29.427946 (+ 31us) server_negotiation.cc:407] Connection header received
0924 07:13:29.428506 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:29.428515 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:29.428588 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:29.428752 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:29.428952 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.430360 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.431145 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.431742 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.432177 (+ 435us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:29.432590 (+ 413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:29.432595 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:29.432595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:13:29.432828 (+ 233us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:29.432870 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:13:29.432886 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:29.432952 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:29.433130 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 07:13:29.433230 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":97,"threads_started":1}
I20260924 07:13:29.433432 19039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:13:29.435159 19039 catalog_manager.cc:1347] Generated new cluster ID: f092b1e28a7f4bef89c4c650a67656c0
I20260924 07:13:29.435209 19039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:13:29.442094 19039 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:13:29.442730 19039 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:13:29.448664 19039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7aee4cd6b02f4ae29cbc18073cda7345: Generated new TSK 0
I20260924 07:13:29.448855 19039 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:13:29.453974 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32975
--builtin_ntp_servers=127.0.98.20:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 07:13:29.534631 19052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:29.534859 19052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:29.534895 19052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:29.536021 19052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:29.536083 19052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:29.536126 19052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:13:29.537523 19052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:29.537763 19052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:29.538091 19052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:29.544065 19057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:29.544404 19058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:29.544363 19052 server_base.cc:1034] running on GCE node
W20260924 07:13:29.544623 19060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:29.544868 19052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:29.545141 19052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:29.546309 19052 hybrid_clock.cc:648] HybridClock initialized: now 1790234009546296 us; error 29 us; skew 500 ppm
I20260924 07:13:29.547561 19052 webserver.cc:460] Webserver started at http://127.0.98.1:46811/ using document root <none> and password file <none>
I20260924 07:13:29.547785 19052 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:29.547852 19052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:29.548018 19052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:29.549113 19052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-0/data/instance:
uuid: "28f57a4412a044cd86285881565648a1"
format_stamp: "Formatted at 2026-09-24 07:13:29 on dist-test-slave-14ww"
server_key: "e0bc6c6c6def48d5bdfa642d4be09def"
server_key_iv: "caa811c9f9d3602abff5fb21c9309d15"
server_key_version: "encryptionkey@0"
I20260924 07:13:29.549461 19052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-0/wal/instance:
uuid: "28f57a4412a044cd86285881565648a1"
format_stamp: "Formatted at 2026-09-24 07:13:29 on dist-test-slave-14ww"
server_key: "e0bc6c6c6def48d5bdfa642d4be09def"
server_key_iv: "caa811c9f9d3602abff5fb21c9309d15"
server_key_version: "encryptionkey@0"
I20260924 07:13:29.550751 19052 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:13:29.551479 19066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:29.551656 19052 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:13:29.551755 19052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-0/wal
uuid: "28f57a4412a044cd86285881565648a1"
format_stamp: "Formatted at 2026-09-24 07:13:29 on dist-test-slave-14ww"
server_key: "e0bc6c6c6def48d5bdfa642d4be09def"
server_key_iv: "caa811c9f9d3602abff5fb21c9309d15"
server_key_version: "encryptionkey@0"
I20260924 07:13:29.551831 19052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:29.561780 19052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:29.562043 19052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:29.562181 19052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:29.562381 19052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:13:29.562711 19052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:13:29.562747 19052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:29.562767 19052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:13:29.562780 19052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:29.568256 19052 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45411
I20260924 07:13:29.568306 19179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45411 every 8 connection(s)
I20260924 07:13:29.568588 19052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:13:29.573830 19032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:29.569342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37861 (local address 127.0.98.62:32975)
0924 07:13:29.569394 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:29.569397 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:29.569734 (+ 337us) server_negotiation.cc:407] Connection header received
0924 07:13:29.570379 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:29.570380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:29.570392 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:29.570450 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:29.571156 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.571653 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.572615 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.572853 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.572907 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:29.573623 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:29.573624 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:29.573625 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:13:29.573629 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:29.573641 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:13:29.573645 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:29.573692 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:29.573755 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 07:13:29.573774 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 07:13:29.574045 19182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:29.569319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32975 (local address 127.0.98.1:37861)
0924 07:13:29.569610 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:29.569651 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:29.570288 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:29.570569 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:29.570573 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:29.570581 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:29.571064 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:29.571069 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.571733 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.571734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:29.572535 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:29.572537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.572930 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.572931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:29.573050 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:29.573501 (+ 451us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:13:29.573502 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:29.573526 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:13:29.573547 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:29.573694 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:29.573696 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:29.573699 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:13:29.573743 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 07:13:29.573814 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":79,"threads_started":1}
I20260924 07:13:29.574592 19180 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32975
I20260924 07:13:29.574714 19180 heartbeater.cc:464] Registering TS with master...
I20260924 07:13:29.574890 19180 heartbeater.cc:511] Master 127.0.98.62:32975 requested a full tablet report, sending...
I20260924 07:13:29.575429 18992 ts_manager.cc:194] Registered new tserver with Master: 28f57a4412a044cd86285881565648a1 (127.0.98.1:45411)
I20260924 07:13:29.576049 18992 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:37861
I20260924 07:13:29.578487 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 19052
I20260924 07:13:29.578619 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:13:29.578815 392 external_mini_cluster.cc:1428] Setting key ca96464647c562ff97d04e0761cab7c5
I20260924 07:13:29.579701 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32975
--builtin_ntp_servers=127.0.98.20:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 07:13:29.660444 19183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:29.660631 19183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:29.660652 19183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:29.661798 19183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:29.661839 19183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:29.661863 19183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:13:29.663120 19183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:29.663290 19183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:29.663513 19183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:29.668426 19191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:29.668438 19189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:29.668565 19183 server_base.cc:1034] running on GCE node
W20260924 07:13:29.668439 19188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:29.668983 19183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:29.669239 19183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:29.670410 19183 hybrid_clock.cc:648] HybridClock initialized: now 1790234009670395 us; error 26 us; skew 500 ppm
I20260924 07:13:29.671715 19183 webserver.cc:460] Webserver started at http://127.0.98.2:39311/ using document root <none> and password file <none>
I20260924 07:13:29.671984 19183 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:29.672060 19183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:29.672214 19183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:29.673295 19183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-1/data/instance:
uuid: "8d646bbb2a6c4389adb444beb1cb3bbf"
format_stamp: "Formatted at 2026-09-24 07:13:29 on dist-test-slave-14ww"
server_key: "41902e3c3633651872f4ef0975a86d49"
server_key_iv: "63824ff484244a72b846bbc32fb291a2"
server_key_version: "encryptionkey@0"
I20260924 07:13:29.673650 19183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8d646bbb2a6c4389adb444beb1cb3bbf"
format_stamp: "Formatted at 2026-09-24 07:13:29 on dist-test-slave-14ww"
server_key: "41902e3c3633651872f4ef0975a86d49"
server_key_iv: "63824ff484244a72b846bbc32fb291a2"
server_key_version: "encryptionkey@0"
I20260924 07:13:29.674901 19183 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:13:29.675607 19197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:29.675767 19183 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:13:29.675858 19183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-1/wal
uuid: "8d646bbb2a6c4389adb444beb1cb3bbf"
format_stamp: "Formatted at 2026-09-24 07:13:29 on dist-test-slave-14ww"
server_key: "41902e3c3633651872f4ef0975a86d49"
server_key_iv: "63824ff484244a72b846bbc32fb291a2"
server_key_version: "encryptionkey@0"
I20260924 07:13:29.675940 19183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:29.689465 19183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:29.689729 19183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:29.689881 19183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:29.690090 19183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:13:29.690387 19183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:13:29.690439 19183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:29.690479 19183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:13:29.690507 19183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:29.696187 19183 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37239
I20260924 07:13:29.696224 19310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37239 every 8 connection(s)
I20260924 07:13:29.696522 19183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:13:29.701337 19032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:29.697243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35067 (local address 127.0.98.62:32975)
0924 07:13:29.697293 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:29.697295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:29.697555 (+ 260us) server_negotiation.cc:407] Connection header received
0924 07:13:29.697987 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:29.697988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:29.697997 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:29.698050 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:29.698654 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.699129 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.700060 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.700248 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.700299 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:29.701120 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:29.701121 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:29.701122 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:13:29.701125 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:29.701140 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:13:29.701143 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:29.701195 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:29.701260 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 07:13:29.701281 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 07:13:29.701520 19313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:29.697239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32975 (local address 127.0.98.2:35067)
0924 07:13:29.697441 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:29.697472 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:29.697904 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:29.698098 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:29.698101 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:29.698109 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:29.698585 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:29.698590 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.699211 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.699211 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:29.699987 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:29.699989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.700318 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.700319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:29.700431 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:29.700986 (+ 555us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:13:29.700987 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:29.701013 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:13:29.701034 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:29.701198 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:29.701200 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:29.701202 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 07:13:29.701248 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 07:13:29.701317 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":81,"threads_started":1}
I20260924 07:13:29.702052 19311 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32975
I20260924 07:13:29.702154 19311 heartbeater.cc:464] Registering TS with master...
I20260924 07:13:29.702327 19311 heartbeater.cc:511] Master 127.0.98.62:32975 requested a full tablet report, sending...
I20260924 07:13:29.702673 18992 ts_manager.cc:194] Registered new tserver with Master: 8d646bbb2a6c4389adb444beb1cb3bbf (127.0.98.2:37239)
I20260924 07:13:29.703048 18992 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:35067
I20260924 07:13:29.704387 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 19183
I20260924 07:13:29.704502 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:13:29.704711 392 external_mini_cluster.cc:1428] Setting key 6bba04161c194f3258dec5235f824763
I20260924 07:13:29.705576 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32975
--builtin_ntp_servers=127.0.98.20:45759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 07:13:29.784677 19314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:29.784881 19314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:29.784904 19314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:29.786026 19314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:29.786067 19314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:29.786090 19314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:13:29.787416 19314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:29.787649 19314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:29.787930 19314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:29.792718 19322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:29.792738 19319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:29.792838 19320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:29.792987 19314 server_base.cc:1034] running on GCE node
I20260924 07:13:29.793187 19314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:29.793385 19314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:29.794523 19314 hybrid_clock.cc:648] HybridClock initialized: now 1790234009794494 us; error 45 us; skew 500 ppm
I20260924 07:13:29.795797 19314 webserver.cc:460] Webserver started at http://127.0.98.3:35833/ using document root <none> and password file <none>
I20260924 07:13:29.796066 19314 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:29.796113 19314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:29.796274 19314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:29.797334 19314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-2/data/instance:
uuid: "4e7bc6128444458bb1ab54a28f55511a"
format_stamp: "Formatted at 2026-09-24 07:13:29 on dist-test-slave-14ww"
server_key: "924d80d3dd108188b83330dfae1aa6dc"
server_key_iv: "0e83911d083d94f9f39b299424544c31"
server_key_version: "encryptionkey@0"
I20260924 07:13:29.797667 19314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4e7bc6128444458bb1ab54a28f55511a"
format_stamp: "Formatted at 2026-09-24 07:13:29 on dist-test-slave-14ww"
server_key: "924d80d3dd108188b83330dfae1aa6dc"
server_key_iv: "0e83911d083d94f9f39b299424544c31"
server_key_version: "encryptionkey@0"
I20260924 07:13:29.798895 19314 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 07:13:29.799649 19328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:29.799854 19314 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 07:13:29.799968 19314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-2/wal
uuid: "4e7bc6128444458bb1ab54a28f55511a"
format_stamp: "Formatted at 2026-09-24 07:13:29 on dist-test-slave-14ww"
server_key: "924d80d3dd108188b83330dfae1aa6dc"
server_key_iv: "0e83911d083d94f9f39b299424544c31"
server_key_version: "encryptionkey@0"
I20260924 07:13:29.800036 19314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:29.835964 19314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:29.836270 19314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:29.836421 19314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:29.836645 19314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:13:29.836994 19314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:13:29.837029 19314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:29.837051 19314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:13:29.837096 19314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:29.842823 19314 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39197
I20260924 07:13:29.842908 19441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39197 every 8 connection(s)
I20260924 07:13:29.843273 19314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:13:29.848481 19032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:29.844058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34445 (local address 127.0.98.62:32975)
0924 07:13:29.844141 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:29.844143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:29.844296 (+ 153us) server_negotiation.cc:407] Connection header received
0924 07:13:29.845122 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:29.845124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:29.845135 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:29.845194 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:29.845821 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.846328 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.847250 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.847433 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.847488 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:29.848253 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:29.848253 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:29.848254 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:13:29.848258 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:29.848270 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:13:29.848273 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:29.848315 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:29.848403 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 07:13:29.848424 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 07:13:29.848702 19444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:29.843992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32975 (local address 127.0.98.3:34445)
0924 07:13:29.844182 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:29.844221 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:29.844977 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:29.845249 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:29.845253 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:29.845262 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:29.845744 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:29.845748 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.846406 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.846407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:29.847176 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:29.847178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.847523 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.847524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:29.847638 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:29.848124 (+ 486us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:13:29.848124 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:29.848150 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:13:29.848171 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:29.848334 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:29.848337 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:29.848339 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 07:13:29.848392 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 07:13:29.848466 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":116,"threads_started":1}
I20260924 07:13:29.849222 19442 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32975
I20260924 07:13:29.849321 19442 heartbeater.cc:464] Registering TS with master...
I20260924 07:13:29.849493 19442 heartbeater.cc:511] Master 127.0.98.62:32975 requested a full tablet report, sending...
I20260924 07:13:29.849877 18992 ts_manager.cc:194] Registered new tserver with Master: 4e7bc6128444458bb1ab54a28f55511a (127.0.98.3:39197)
I20260924 07:13:29.850297 18992 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:34445
I20260924 07:13:29.850394 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 19314
I20260924 07:13:29.850492 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:13:29.850694 392 external_mini_cluster.cc:1428] Setting key b867aaf9f73aaba292191af584308cf6
I20260924 07:13:29.851598 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:13:29.855552 19032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:29.852945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57462 (local address 127.0.98.62:32975)
0924 07:13:29.852983 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:29.852985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:29.853114 (+ 129us) server_negotiation.cc:407] Connection header received
0924 07:13:29.853152 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:29.853153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:29.853160 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:29.853209 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:29.853326 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.853829 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.854531 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.854697 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.854745 (+ 48us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:29.855359 (+ 614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:29.855361 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:29.855361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:13:29.855365 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:29.855375 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:13:29.855379 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:29.855450 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:29.855498 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 07:13:29.855506 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 07:13:29.856928 18992 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57462:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:13:29.858459 18992 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:13:29.864550 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:29.860913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37239 (local address 127.0.0.1:60668)
0924 07:13:29.861049 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:29.861059 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:29.861122 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:29.861443 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:29.861444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:29.861446 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:29.861502 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:29.861504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.862384 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.862385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:29.863464 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:29.863467 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.864264 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.864265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:29.864339 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:29.864351 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 07:13:29.864430 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 07:13:29.864479 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":68,"threads_started":1}
I20260924 07:13:29.864562 19455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:29.861030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60668 (local address 127.0.98.2:37239)
0924 07:13:29.861241 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:29.861246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:29.861255 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:13:29.861292 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:29.861294 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:29.861307 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:29.861379 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:29.861581 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.862297 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.863540 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.864160 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.864258 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:29.864348 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:29.864445 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 07:13:29.864483 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":78,"threads_started":1}
I20260924 07:13:29.865114 19450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:29.860655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39197 (local address 127.0.0.1:54886)
0924 07:13:29.861011 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:29.861052 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:29.861146 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:29.861382 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:29.861385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:29.861387 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:29.861454 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:29.861456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.862714 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.862715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:29.863740 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:29.863742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.864887 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.864888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:29.864941 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:29.864944 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:13:29.865011 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 07:13:29.865046 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":22,"thread_start_us":58,"threads_started":1}
I20260924 07:13:29.865125 19453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:29.860805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60934 (local address 127.0.98.1:45411)
0924 07:13:29.860984 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:29.860987 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:29.861115 (+ 128us) server_negotiation.cc:407] Connection header received
0924 07:13:29.861455 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:29.861458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:29.861474 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:29.861552 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:29.861740 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.862792 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.863948 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.864632 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.864671 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:29.864768 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:29.864997 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 07:13:29.865042 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":102,"threads_started":1}
I20260924 07:13:29.865154 19452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:29.860753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54886 (local address 127.0.98.3:39197)
0924 07:13:29.860911 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:29.860915 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:29.861116 (+ 201us) server_negotiation.cc:407] Connection header received
0924 07:13:29.861275 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:29.861277 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:29.861294 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:29.861376 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:29.861549 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.862616 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.863831 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.864547 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.864620 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:29.864698 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:29.865040 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 07:13:29.865082 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":67,"threads_started":1}
I20260924 07:13:29.865387 19245 tablet_service.cc:1467] Processing CreateTablet for tablet a9d4149d30614ecda83b0e86d38d9534 (DEFAULT_TABLE table=test-table [id=10d71027d8de409ba6ceee8675901acb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:13:29.865377 19454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:29.860686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45411 (local address 127.0.0.1:60934)
0924 07:13:29.861031 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:29.861052 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:29.861146 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:29.861621 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:29.861622 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:29.861624 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:29.861653 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:29.861654 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.862880 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.862881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:29.863847 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:29.863849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.864836 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.864837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:29.864905 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:29.864908 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:13:29.865271 (+ 363us) client_negotiation.cc:240] Negotiation successful
0924 07:13:29.865320 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":104,"threads_started":1}
I20260924 07:13:29.865800 19245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9d4149d30614ecda83b0e86d38d9534. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:29.866145 19376 tablet_service.cc:1467] Processing CreateTablet for tablet a9d4149d30614ecda83b0e86d38d9534 (DEFAULT_TABLE table=test-table [id=10d71027d8de409ba6ceee8675901acb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:13:29.866466 19376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9d4149d30614ecda83b0e86d38d9534. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:29.866564 19114 tablet_service.cc:1467] Processing CreateTablet for tablet a9d4149d30614ecda83b0e86d38d9534 (DEFAULT_TABLE table=test-table [id=10d71027d8de409ba6ceee8675901acb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:13:29.866883 19114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9d4149d30614ecda83b0e86d38d9534. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:29.869227 19456 tablet_bootstrap.cc:492] T a9d4149d30614ecda83b0e86d38d9534 P 8d646bbb2a6c4389adb444beb1cb3bbf: Bootstrap starting.
I20260924 07:13:29.869534 19457 tablet_bootstrap.cc:492] T a9d4149d30614ecda83b0e86d38d9534 P 28f57a4412a044cd86285881565648a1: Bootstrap starting.
I20260924 07:13:29.869829 19458 tablet_bootstrap.cc:492] T a9d4149d30614ecda83b0e86d38d9534 P 4e7bc6128444458bb1ab54a28f55511a: Bootstrap starting.
I20260924 07:13:29.869990 19456 tablet_bootstrap.cc:654] T a9d4149d30614ecda83b0e86d38d9534 P 8d646bbb2a6c4389adb444beb1cb3bbf: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:29.870203 19457 tablet_bootstrap.cc:654] T a9d4149d30614ecda83b0e86d38d9534 P 28f57a4412a044cd86285881565648a1: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:29.870299 19456 log.cc:824] T a9d4149d30614ecda83b0e86d38d9534 P 8d646bbb2a6c4389adb444beb1cb3bbf: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:29.870448 19458 tablet_bootstrap.cc:654] T a9d4149d30614ecda83b0e86d38d9534 P 4e7bc6128444458bb1ab54a28f55511a: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:29.870515 19457 log.cc:824] T a9d4149d30614ecda83b0e86d38d9534 P 28f57a4412a044cd86285881565648a1: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:29.870802 19458 log.cc:824] T a9d4149d30614ecda83b0e86d38d9534 P 4e7bc6128444458bb1ab54a28f55511a: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:29.871146 19456 tablet_bootstrap.cc:492] T a9d4149d30614ecda83b0e86d38d9534 P 8d646bbb2a6c4389adb444beb1cb3bbf: No bootstrap required, opened a new log
I20260924 07:13:29.871223 19456 ts_tablet_manager.cc:1397] T a9d4149d30614ecda83b0e86d38d9534 P 8d646bbb2a6c4389adb444beb1cb3bbf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:13:29.871392 19457 tablet_bootstrap.cc:492] T a9d4149d30614ecda83b0e86d38d9534 P 28f57a4412a044cd86285881565648a1: No bootstrap required, opened a new log
I20260924 07:13:29.871465 19458 tablet_bootstrap.cc:492] T a9d4149d30614ecda83b0e86d38d9534 P 4e7bc6128444458bb1ab54a28f55511a: No bootstrap required, opened a new log
I20260924 07:13:29.871531 19458 ts_tablet_manager.cc:1397] T a9d4149d30614ecda83b0e86d38d9534 P 4e7bc6128444458bb1ab54a28f55511a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:13:29.871601 19457 ts_tablet_manager.cc:1397] T a9d4149d30614ecda83b0e86d38d9534 P 28f57a4412a044cd86285881565648a1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:13:29.873668 19457 raft_consensus.cc:348] T a9d4149d30614ecda83b0e86d38d9534 P 28f57a4412a044cd86285881565648a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7bc6128444458bb1ab54a28f55511a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39197 } } peers { permanent_uuid: "28f57a4412a044cd86285881565648a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45411 } } peers { permanent_uuid: "8d646bbb2a6c4389adb444beb1cb3bbf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37239 } }
I20260924 07:13:29.873821 19457 raft_consensus.cc:374] T a9d4149d30614ecda83b0e86d38d9534 P 28f57a4412a044cd86285881565648a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:29.873864 19457 raft_consensus.cc:729] T a9d4149d30614ecda83b0e86d38d9534 P 28f57a4412a044cd86285881565648a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28f57a4412a044cd86285881565648a1, State: Initialized, Role: FOLLOWER
I20260924 07:13:29.873979 19457 consensus_queue.cc:260] T a9d4149d30614ecda83b0e86d38d9534 P 28f57a4412a044cd86285881565648a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7bc6128444458bb1ab54a28f55511a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39197 } } peers { permanent_uuid: "28f57a4412a044cd86285881565648a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45411 } } peers { permanent_uuid: "8d646bbb2a6c4389adb444beb1cb3bbf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37239 } }
I20260924 07:13:29.873879 19456 raft_consensus.cc:348] T a9d4149d30614ecda83b0e86d38d9534 P 8d646bbb2a6c4389adb444beb1cb3bbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7bc6128444458bb1ab54a28f55511a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39197 } } peers { permanent_uuid: "28f57a4412a044cd86285881565648a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45411 } } peers { permanent_uuid: "8d646bbb2a6c4389adb444beb1cb3bbf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37239 } }
I20260924 07:13:29.874105 19456 raft_consensus.cc:374] T a9d4149d30614ecda83b0e86d38d9534 P 8d646bbb2a6c4389adb444beb1cb3bbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:29.874044 19458 raft_consensus.cc:348] T a9d4149d30614ecda83b0e86d38d9534 P 4e7bc6128444458bb1ab54a28f55511a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7bc6128444458bb1ab54a28f55511a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39197 } } peers { permanent_uuid: "28f57a4412a044cd86285881565648a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45411 } } peers { permanent_uuid: "8d646bbb2a6c4389adb444beb1cb3bbf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37239 } }
I20260924 07:13:29.874141 19456 raft_consensus.cc:729] T a9d4149d30614ecda83b0e86d38d9534 P 8d646bbb2a6c4389adb444beb1cb3bbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d646bbb2a6c4389adb444beb1cb3bbf, State: Initialized, Role: FOLLOWER
I20260924 07:13:29.874178 19458 raft_consensus.cc:374] T a9d4149d30614ecda83b0e86d38d9534 P 4e7bc6128444458bb1ab54a28f55511a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:29.874213 19458 raft_consensus.cc:729] T a9d4149d30614ecda83b0e86d38d9534 P 4e7bc6128444458bb1ab54a28f55511a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e7bc6128444458bb1ab54a28f55511a, State: Initialized, Role: FOLLOWER
I20260924 07:13:29.874236 19457 ts_tablet_manager.cc:1428] T a9d4149d30614ecda83b0e86d38d9534 P 28f57a4412a044cd86285881565648a1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 07:13:29.874264 19456 consensus_queue.cc:260] T a9d4149d30614ecda83b0e86d38d9534 P 8d646bbb2a6c4389adb444beb1cb3bbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7bc6128444458bb1ab54a28f55511a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39197 } } peers { permanent_uuid: "28f57a4412a044cd86285881565648a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45411 } } peers { permanent_uuid: "8d646bbb2a6c4389adb444beb1cb3bbf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37239 } }
I20260924 07:13:29.874367 19180 heartbeater.cc:503] Master 127.0.98.62:32975 was elected leader, sending a full tablet report...
I20260924 07:13:29.874527 19456 ts_tablet_manager.cc:1428] T a9d4149d30614ecda83b0e86d38d9534 P 8d646bbb2a6c4389adb444beb1cb3bbf: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260924 07:13:29.874583 19311 heartbeater.cc:503] Master 127.0.98.62:32975 was elected leader, sending a full tablet report...
I20260924 07:13:29.874701 19458 consensus_queue.cc:260] T a9d4149d30614ecda83b0e86d38d9534 P 4e7bc6128444458bb1ab54a28f55511a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7bc6128444458bb1ab54a28f55511a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39197 } } peers { permanent_uuid: "28f57a4412a044cd86285881565648a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45411 } } peers { permanent_uuid: "8d646bbb2a6c4389adb444beb1cb3bbf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37239 } }
I20260924 07:13:29.874969 19458 ts_tablet_manager.cc:1428] T a9d4149d30614ecda83b0e86d38d9534 P 4e7bc6128444458bb1ab54a28f55511a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 07:13:29.874990 19442 heartbeater.cc:503] Master 127.0.98.62:32975 was elected leader, sending a full tablet report...
I20260924 07:13:29.913285 19462 raft_consensus.cc:482] T a9d4149d30614ecda83b0e86d38d9534 P 28f57a4412a044cd86285881565648a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:13:29.913385 19462 raft_consensus.cc:504] T a9d4149d30614ecda83b0e86d38d9534 P 28f57a4412a044cd86285881565648a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7bc6128444458bb1ab54a28f55511a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39197 } } peers { permanent_uuid: "28f57a4412a044cd86285881565648a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45411 } } peers { permanent_uuid: "8d646bbb2a6c4389adb444beb1cb3bbf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37239 } }
I20260924 07:13:29.913787 19462 leader_election.cc:288] T a9d4149d30614ecda83b0e86d38d9534 P 28f57a4412a044cd86285881565648a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e7bc6128444458bb1ab54a28f55511a (127.0.98.3:39197), 8d646bbb2a6c4389adb444beb1cb3bbf (127.0.98.2:37239)
I20260924 07:13:29.916726 19455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:29.914022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56165 (local address 127.0.98.2:37239)
0924 07:13:29.914065 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:29.914067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:29.914078 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:13:29.914159 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:29.914161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:29.914173 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:29.914244 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:29.914415 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.914932 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.916006 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.916358 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.916449 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:29.916459 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:29.916618 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 07:13:29.916659 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 07:13:29.916752 19182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:29.913958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37239 (local address 127.0.98.1:56165)
0924 07:13:29.914022 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:29.914036 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:29.914084 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:29.914247 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:29.914248 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:29.914249 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:29.914316 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:29.914318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.915021 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.915022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:29.915909 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:29.915912 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.916450 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.916451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:29.916523 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:29.916526 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:13:29.916609 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 07:13:29.916650 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 07:13:29.916766 19452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:29.914046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54929 (local address 127.0.98.3:39197)
0924 07:13:29.914088 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:29.914090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:29.914225 (+ 135us) server_negotiation.cc:407] Connection header received
0924 07:13:29.914294 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:29.914295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:29.914303 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:29.914371 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:29.914532 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.915006 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.916045 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.916388 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.916476 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:29.916484 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:29.916644 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 07:13:29.916679 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 07:13:29.916800 19465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:29.913984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39197 (local address 127.0.98.1:54929)
0924 07:13:29.914146 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:29.914161 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:29.914236 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:29.914373 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:29.914373 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:29.914375 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:29.914442 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:29.914444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.915090 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.915091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:29.915959 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:29.915961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.916483 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.916484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:29.916564 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:29.916573 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 07:13:29.916639 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 07:13:29.916677 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":100,"threads_started":1}
I20260924 07:13:29.917155 19265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9d4149d30614ecda83b0e86d38d9534" candidate_uuid: "28f57a4412a044cd86285881565648a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d646bbb2a6c4389adb444beb1cb3bbf" is_pre_election: true
I20260924 07:13:29.917271 19265 raft_consensus.cc:2454] T a9d4149d30614ecda83b0e86d38d9534 P 8d646bbb2a6c4389adb444beb1cb3bbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28f57a4412a044cd86285881565648a1 in term 0.
I20260924 07:13:29.917202 19396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9d4149d30614ecda83b0e86d38d9534" candidate_uuid: "28f57a4412a044cd86285881565648a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e7bc6128444458bb1ab54a28f55511a" is_pre_election: true
I20260924 07:13:29.917378 19396 raft_consensus.cc:2454] T a9d4149d30614ecda83b0e86d38d9534 P 4e7bc6128444458bb1ab54a28f55511a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28f57a4412a044cd86285881565648a1 in term 0.
I20260924 07:13:29.917512 19068 leader_election.cc:302] T a9d4149d30614ecda83b0e86d38d9534 P 28f57a4412a044cd86285881565648a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28f57a4412a044cd86285881565648a1, 8d646bbb2a6c4389adb444beb1cb3bbf; no voters:
I20260924 07:13:29.917629 19462 raft_consensus.cc:2781] T a9d4149d30614ecda83b0e86d38d9534 P 28f57a4412a044cd86285881565648a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:13:29.917685 19462 raft_consensus.cc:482] T a9d4149d30614ecda83b0e86d38d9534 P 28f57a4412a044cd86285881565648a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:13:29.917711 19462 raft_consensus.cc:3037] T a9d4149d30614ecda83b0e86d38d9534 P 28f57a4412a044cd86285881565648a1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:29.918486 19462 raft_consensus.cc:504] T a9d4149d30614ecda83b0e86d38d9534 P 28f57a4412a044cd86285881565648a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7bc6128444458bb1ab54a28f55511a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39197 } } peers { permanent_uuid: "28f57a4412a044cd86285881565648a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45411 } } peers { permanent_uuid: "8d646bbb2a6c4389adb444beb1cb3bbf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37239 } }
I20260924 07:13:29.918615 19462 leader_election.cc:288] T a9d4149d30614ecda83b0e86d38d9534 P 28f57a4412a044cd86285881565648a1 [CANDIDATE]: Term 1 election: Requested vote from peers 4e7bc6128444458bb1ab54a28f55511a (127.0.98.3:39197), 8d646bbb2a6c4389adb444beb1cb3bbf (127.0.98.2:37239)
I20260924 07:13:29.918776 19265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9d4149d30614ecda83b0e86d38d9534" candidate_uuid: "28f57a4412a044cd86285881565648a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d646bbb2a6c4389adb444beb1cb3bbf"
I20260924 07:13:29.918776 19396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9d4149d30614ecda83b0e86d38d9534" candidate_uuid: "28f57a4412a044cd86285881565648a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e7bc6128444458bb1ab54a28f55511a"
I20260924 07:13:29.918856 19396 raft_consensus.cc:3037] T a9d4149d30614ecda83b0e86d38d9534 P 4e7bc6128444458bb1ab54a28f55511a [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:29.918857 19265 raft_consensus.cc:3037] T a9d4149d30614ecda83b0e86d38d9534 P 8d646bbb2a6c4389adb444beb1cb3bbf [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:29.919736 19265 raft_consensus.cc:2454] T a9d4149d30614ecda83b0e86d38d9534 P 8d646bbb2a6c4389adb444beb1cb3bbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28f57a4412a044cd86285881565648a1 in term 1.
I20260924 07:13:29.919736 19396 raft_consensus.cc:2454] T a9d4149d30614ecda83b0e86d38d9534 P 4e7bc6128444458bb1ab54a28f55511a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28f57a4412a044cd86285881565648a1 in term 1.
I20260924 07:13:29.919900 19068 leader_election.cc:302] T a9d4149d30614ecda83b0e86d38d9534 P 28f57a4412a044cd86285881565648a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28f57a4412a044cd86285881565648a1, 8d646bbb2a6c4389adb444beb1cb3bbf; no voters:
I20260924 07:13:29.920058 19462 raft_consensus.cc:2781] T a9d4149d30614ecda83b0e86d38d9534 P 28f57a4412a044cd86285881565648a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:13:29.920179 19462 raft_consensus.cc:686] T a9d4149d30614ecda83b0e86d38d9534 P 28f57a4412a044cd86285881565648a1 [term 1 LEADER]: Becoming Leader. State: Replica: 28f57a4412a044cd86285881565648a1, State: Running, Role: LEADER
I20260924 07:13:29.920269 19462 consensus_queue.cc:237] T a9d4149d30614ecda83b0e86d38d9534 P 28f57a4412a044cd86285881565648a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7bc6128444458bb1ab54a28f55511a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39197 } } peers { permanent_uuid: "28f57a4412a044cd86285881565648a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45411 } } peers { permanent_uuid: "8d646bbb2a6c4389adb444beb1cb3bbf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37239 } }
I20260924 07:13:29.921048 18990 catalog_manager.cc:5495] T a9d4149d30614ecda83b0e86d38d9534 P 28f57a4412a044cd86285881565648a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28f57a4412a044cd86285881565648a1 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "28f57a4412a044cd86285881565648a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7bc6128444458bb1ab54a28f55511a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28f57a4412a044cd86285881565648a1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d646bbb2a6c4389adb444beb1cb3bbf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37239 } health_report { overall_health: UNKNOWN } } }
I20260924 07:13:29.937098 19453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:29.934939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60938 (local address 127.0.98.1:45411)
0924 07:13:29.934985 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:29.934986 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:29.934994 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:13:29.935026 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:29.935026 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:29.935041 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:29.935098 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:13:29.935192 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.935621 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.936310 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:29.936490 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:29.936559 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:29.936682 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:13:29.936825 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:13:29.936898 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:29.936965 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 07:13:29.936973 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 07:13:29.938786 19265 raft_consensus.cc:1261] T a9d4149d30614ecda83b0e86d38d9534 P 8d646bbb2a6c4389adb444beb1cb3bbf [term 1 FOLLOWER]: Refusing update from remote peer 28f57a4412a044cd86285881565648a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:13:29.938784 19396 raft_consensus.cc:1261] T a9d4149d30614ecda83b0e86d38d9534 P 4e7bc6128444458bb1ab54a28f55511a [term 1 FOLLOWER]: Refusing update from remote peer 28f57a4412a044cd86285881565648a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:13:29.939117 19466 consensus_queue.cc:1035] T a9d4149d30614ecda83b0e86d38d9534 P 28f57a4412a044cd86285881565648a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e7bc6128444458bb1ab54a28f55511a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:13:29.939231 19462 consensus_queue.cc:1035] T a9d4149d30614ecda83b0e86d38d9534 P 28f57a4412a044cd86285881565648a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d646bbb2a6c4389adb444beb1cb3bbf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:13:29.941630 19470 mvcc.cc:205] Tried to move back new op lower bound from 7332798504706478080 to 7332798504634011648. Current Snapshot: MvccSnapshot[applied={T|T < 7332798504706478080}]
I20260924 07:13:29.943430 18990 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:13:29.943516 18990 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:13:29.944257 18990 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7aee4cd6b02f4ae29cbc18073cda7345: Sending DeleteTablet for 3 replicas of tablet a9d4149d30614ecda83b0e86d38d9534
I20260924 07:13:29.944548 19376 tablet_service.cc:1514] Processing DeleteTablet for tablet a9d4149d30614ecda83b0e86d38d9534 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:13:29 UTC) from {username='slave'} at 127.0.0.1:54886
I20260924 07:13:29.944666 19245 tablet_service.cc:1514] Processing DeleteTablet for tablet a9d4149d30614ecda83b0e86d38d9534 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:13:29 UTC) from {username='slave'} at 127.0.0.1:60668
I20260924 07:13:29.944666 19114 tablet_service.cc:1514] Processing DeleteTablet for tablet a9d4149d30614ecda83b0e86d38d9534 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:13:29 UTC) from {username='slave'} at 127.0.0.1:60934
I20260924 07:13:29.944885 19479 tablet_replica.cc:330] T a9d4149d30614ecda83b0e86d38d9534 P 4e7bc6128444458bb1ab54a28f55511a: stopping tablet replica
I20260924 07:13:29.945013 19479 raft_consensus.cc:2229] T a9d4149d30614ecda83b0e86d38d9534 P 4e7bc6128444458bb1ab54a28f55511a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:13:29.945066 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 19052
I20260924 07:13:29.945765 19479 raft_consensus.cc:2258] T a9d4149d30614ecda83b0e86d38d9534 P 4e7bc6128444458bb1ab54a28f55511a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:13:29.946242 19479 ts_tablet_manager.cc:1905] T a9d4149d30614ecda83b0e86d38d9534 P 4e7bc6128444458bb1ab54a28f55511a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:13:29.948436 19479 ts_tablet_manager.cc:1918] T a9d4149d30614ecda83b0e86d38d9534 P 4e7bc6128444458bb1ab54a28f55511a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:13:29.948509 19479 log.cc:1196] T a9d4149d30614ecda83b0e86d38d9534 P 4e7bc6128444458bb1ab54a28f55511a: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790233772407728-392-0/minicluster-data/ts-2/wal/wals/a9d4149d30614ecda83b0e86d38d9534
I20260924 07:13:29.948774 19479 ts_tablet_manager.cc:1939] T a9d4149d30614ecda83b0e86d38d9534 P 4e7bc6128444458bb1ab54a28f55511a: Deleting consensus metadata
I20260924 07:13:29.949158 18979 catalog_manager.cc:4841] TS 4e7bc6128444458bb1ab54a28f55511a (127.0.98.3:39197): tablet a9d4149d30614ecda83b0e86d38d9534 (table test-table [id=10d71027d8de409ba6ceee8675901acb]) successfully deleted
I20260924 07:13:29.952852 19481 tablet_replica.cc:330] T a9d4149d30614ecda83b0e86d38d9534 P 8d646bbb2a6c4389adb444beb1cb3bbf: stopping tablet replica
I20260924 07:13:29.953017 19481 raft_consensus.cc:2229] T a9d4149d30614ecda83b0e86d38d9534 P 8d646bbb2a6c4389adb444beb1cb3bbf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:13:29.953119 19481 raft_consensus.cc:2258] T a9d4149d30614ecda83b0e86d38d9534 P 8d646bbb2a6c4389adb444beb1cb3bbf [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 07:13:29.953737 18977 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45411 (error 108)
I20260924 07:13:29.953939 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 19183
I20260924 07:13:29.954066 19481 ts_tablet_manager.cc:1905] T a9d4149d30614ecda83b0e86d38d9534 P 8d646bbb2a6c4389adb444beb1cb3bbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:13:29.954715 19454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:29.954589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45411 (local address 127.0.0.1:60954)
0924 07:13:29.954636 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:29.954675 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45411: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 07:13:29.954849 18977 catalog_manager.cc:4584] TS 28f57a4412a044cd86285881565648a1 (127.0.98.1:45411): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a9d4149d30614ecda83b0e86d38d9534: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45411: connect: Connection refused (error 111)
I20260924 07:13:29.958972 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 19314
I20260924 07:13:29.959043 19454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:29.958934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37239 (local address 127.0.0.1:60680)
0924 07:13:29.958996 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:29.959021 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:37239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 07:13:29.963985 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 18961
2026-09-24T07:13:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (677 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:13:30 dist-test-slave-14ww krb5kdc[19485](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:13:30 dist-test-slave-14ww krb5kdc[19485](info): set up 2 sockets
Sep 24 07:13:30 dist-test-slave-14ww krb5kdc[19485](info): commencing operation
krb5kdc: starting...
W20260924 07:13:30.996482 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:13:31 dist-test-slave-14ww krb5kdc[19485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:13:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:13:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:13:31.113762 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44379
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44379
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:31.192770 19498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:31.192999 19498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:31.193034 19498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:31.194181 19498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:13:31.194250 19498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:31.194274 19498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:31.194295 19498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:13:31.194324 19498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:13:31.195739 19498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:31.196005 19498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:31.196220 19498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:31.201087 19506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:31.201150 19504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:31.201086 19503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:31.201304 19498 server_base.cc:1034] running on GCE node
I20260924 07:13:31.201552 19498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:31.201838 19498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:31.203004 19498 hybrid_clock.cc:648] HybridClock initialized: now 1790234011202992 us; error 31 us; skew 500 ppm
Sep 24 07:13:31 dist-test-slave-14ww krb5kdc[19485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:31.205462 19498 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:13:31.205870 19498 webserver.cc:460] Webserver started at http://127.0.98.62:45849/ using document root <none> and password file <none>
I20260924 07:13:31.206113 19498 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:31.206183 19498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:31.206333 19498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:31.207370 19498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "72f253a9400f4868a2041067ef79f1c8"
format_stamp: "Formatted at 2026-09-24 07:13:31 on dist-test-slave-14ww"
server_key: "d0fe56c59a097a17f0762c7ed195ec3b"
server_key_iv: "fbd1e80cb77066835efd74c8dca5cd8c"
server_key_version: "encryptionkey@0"
I20260924 07:13:31.207743 19498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "72f253a9400f4868a2041067ef79f1c8"
format_stamp: "Formatted at 2026-09-24 07:13:31 on dist-test-slave-14ww"
server_key: "d0fe56c59a097a17f0762c7ed195ec3b"
server_key_iv: "fbd1e80cb77066835efd74c8dca5cd8c"
server_key_version: "encryptionkey@0"
I20260924 07:13:31.209102 19498 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:13:31.209826 19513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:31.209985 19498 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:13:31.210101 19498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "72f253a9400f4868a2041067ef79f1c8"
format_stamp: "Formatted at 2026-09-24 07:13:31 on dist-test-slave-14ww"
server_key: "d0fe56c59a097a17f0762c7ed195ec3b"
server_key_iv: "fbd1e80cb77066835efd74c8dca5cd8c"
server_key_version: "encryptionkey@0"
I20260924 07:13:31.210177 19498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:31.224715 19498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:31.226568 19498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:31.226722 19498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:31.230959 19498 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44379
I20260924 07:13:31.230993 19565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44379 every 8 connection(s)
I20260924 07:13:31.231341 19498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:13:31.232175 19566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:31.234436 19566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72f253a9400f4868a2041067ef79f1c8: Bootstrap starting.
I20260924 07:13:31.234997 19566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72f253a9400f4868a2041067ef79f1c8: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:31.235276 19566 log.cc:824] T 00000000000000000000000000000000 P 72f253a9400f4868a2041067ef79f1c8: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:31.235960 19566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72f253a9400f4868a2041067ef79f1c8: No bootstrap required, opened a new log
I20260924 07:13:31.237596 19566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72f253a9400f4868a2041067ef79f1c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72f253a9400f4868a2041067ef79f1c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44379 } }
I20260924 07:13:31.237725 19566 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72f253a9400f4868a2041067ef79f1c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:31.237746 19566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72f253a9400f4868a2041067ef79f1c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72f253a9400f4868a2041067ef79f1c8, State: Initialized, Role: FOLLOWER
I20260924 07:13:31.237953 19566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72f253a9400f4868a2041067ef79f1c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72f253a9400f4868a2041067ef79f1c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44379 } }
I20260924 07:13:31.238016 19566 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72f253a9400f4868a2041067ef79f1c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:13:31.238066 19566 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72f253a9400f4868a2041067ef79f1c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:13:31.238025 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 19498
I20260924 07:13:31.238114 19566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72f253a9400f4868a2041067ef79f1c8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:31.238161 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:13:31.238349 392 external_mini_cluster.cc:1428] Setting key fad47cefb023503dda5c0654fbbfc611
I20260924 07:13:31.238726 19566 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72f253a9400f4868a2041067ef79f1c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72f253a9400f4868a2041067ef79f1c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44379 } }
I20260924 07:13:31.238833 19566 leader_election.cc:302] T 00000000000000000000000000000000 P 72f253a9400f4868a2041067ef79f1c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72f253a9400f4868a2041067ef79f1c8; no voters:
I20260924 07:13:31.239004 19566 leader_election.cc:288] T 00000000000000000000000000000000 P 72f253a9400f4868a2041067ef79f1c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:13:31.239045 19571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72f253a9400f4868a2041067ef79f1c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:13:31.239190 19571 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72f253a9400f4868a2041067ef79f1c8 [term 1 LEADER]: Becoming Leader. State: Replica: 72f253a9400f4868a2041067ef79f1c8, State: Running, Role: LEADER
I20260924 07:13:31.239307 19566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72f253a9400f4868a2041067ef79f1c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:13:31.239308 19571 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72f253a9400f4868a2041067ef79f1c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72f253a9400f4868a2041067ef79f1c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44379 } }
I20260924 07:13:31.239880 19572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72f253a9400f4868a2041067ef79f1c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72f253a9400f4868a2041067ef79f1c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72f253a9400f4868a2041067ef79f1c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44379 } } }
I20260924 07:13:31.240146 19572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72f253a9400f4868a2041067ef79f1c8 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:31.240069 19573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72f253a9400f4868a2041067ef79f1c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72f253a9400f4868a2041067ef79f1c8. Latest consensus state: current_term: 1 leader_uuid: "72f253a9400f4868a2041067ef79f1c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72f253a9400f4868a2041067ef79f1c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44379 } } }
I20260924 07:13:31.240373 19573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72f253a9400f4868a2041067ef79f1c8 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:31.240550 19580 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:13:31.241078 19580 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:13:31.242484 19580 catalog_manager.cc:1347] Generated new cluster ID: b0a164c7394c40f0ba278ecc8ab0862e
I20260924 07:13:31.242538 19580 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 07:13:31 dist-test-slave-14ww krb5kdc[19485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:13:31.248859 19570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:31.238824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60898 (local address 127.0.98.62:44379)
0924 07:13:31.239018 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:31.239028 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:31.239062 (+ 34us) server_negotiation.cc:407] Connection header received
0924 07:13:31.239178 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:31.239185 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:31.239240 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:31.240191 (+ 951us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:31.240299 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:31.241850 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:31.242863 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:31.243275 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:31.243439 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:31.245695 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:31.245699 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:31.245700 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:31.245711 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:31.247364 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:31.247733 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:31.247736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:31.247736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:31.247832 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:31.248050 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:31.248051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:31.248051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:31.248265 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:31.248451 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:31.248513 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 07:13:31.248644 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":87,"threads_started":1}
I20260924 07:13:31.256286 19580 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:13:31.256964 19580 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:13:31.260679 19580 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72f253a9400f4868a2041067ef79f1c8: Generated new TSK 0
I20260924 07:13:31.260874 19580 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 07:13:31.293780 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44379
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:31.378647 19592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:31.378836 19592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:31.378859 19592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:31.380019 19592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:31.380060 19592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:31.380087 19592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:13:31.381681 19592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:31.381906 19592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:31.382144 19592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:31.387341 19597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:31.387364 19598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:31.387296 19600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:31.387550 19592 server_base.cc:1034] running on GCE node
I20260924 07:13:31.387778 19592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:31.388022 19592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:31.389190 19592 hybrid_clock.cc:648] HybridClock initialized: now 1790234011389170 us; error 41 us; skew 500 ppm
Sep 24 07:13:31 dist-test-slave-14ww krb5kdc[19485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:31.391566 19592 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 07:13:31.392020 19592 webserver.cc:460] Webserver started at http://127.0.98.1:43805/ using document root <none> and password file <none>
I20260924 07:13:31.392233 19592 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:31.392300 19592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:31.392457 19592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:31.393518 19592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-0/data/instance:
uuid: "d924db38ce074bf080df04b7a84235d8"
format_stamp: "Formatted at 2026-09-24 07:13:31 on dist-test-slave-14ww"
server_key: "598dc198c2d5780c526acf76398f174f"
server_key_iv: "4567e3fd93ff2676de464afbb2c30dcb"
server_key_version: "encryptionkey@0"
I20260924 07:13:31.393872 19592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d924db38ce074bf080df04b7a84235d8"
format_stamp: "Formatted at 2026-09-24 07:13:31 on dist-test-slave-14ww"
server_key: "598dc198c2d5780c526acf76398f174f"
server_key_iv: "4567e3fd93ff2676de464afbb2c30dcb"
server_key_version: "encryptionkey@0"
I20260924 07:13:31.395152 19592 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 07:13:31.395889 19607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:31.396123 19592 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:13:31.396229 19592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-0/wal
uuid: "d924db38ce074bf080df04b7a84235d8"
format_stamp: "Formatted at 2026-09-24 07:13:31 on dist-test-slave-14ww"
server_key: "598dc198c2d5780c526acf76398f174f"
server_key_iv: "4567e3fd93ff2676de464afbb2c30dcb"
server_key_version: "encryptionkey@0"
I20260924 07:13:31.396313 19592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:31.421590 19592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:31.423337 19592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:31.423502 19592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:31.423722 19592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:13:31.424021 19592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:13:31.424078 19592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:31.424109 19592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:13:31.424139 19592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:31.430019 19592 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45967
I20260924 07:13:31.430070 19720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45967 every 8 connection(s)
I20260924 07:13:31.430347 19592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 07:13:31 dist-test-slave-14ww krb5kdc[19485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:13:31.439118 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 19592
I20260924 07:13:31.439244 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:13:31.439471 392 external_mini_cluster.cc:1428] Setting key 73a7ebb2e8ff52267840e55c13a53d65
I20260924 07:13:31.439667 19570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:31.430993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34145 (local address 127.0.98.62:44379)
0924 07:13:31.431057 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:31.431075 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:31.431262 (+ 187us) server_negotiation.cc:407] Connection header received
0924 07:13:31.432240 (+ 978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:31.432241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:31.432257 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:31.432309 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:31.432960 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:31.433470 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:31.434473 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:31.434682 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:31.434767 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:31.436799 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:31.436800 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:31.436800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:31.436806 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:31.438213 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:31.438572 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:31.438572 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:31.438573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:31.438622 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:31.438825 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:31.438826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:31.438826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:31.438993 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:31.439063 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:31.439463 (+ 400us) server_negotiation.cc:299] Negotiation successful
0924 07:13:31.439568 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 07:13:31.439854 19723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:31.430957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44379 (local address 127.0.98.1:34145)
0924 07:13:31.431137 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:31.431179 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:31.432117 (+ 938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:31.432375 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:31.432379 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:31.432621 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:31.432861 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:31.432866 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:31.433578 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:31.433579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:31.434372 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:31.434374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:31.434777 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:31.434778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:31.434912 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:31.435342 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:31.435350 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:31.436665 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:31.438292 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:31.438298 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:31.438299 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:31.438506 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:31.438681 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:31.438681 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:31.438682 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:31.438770 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:31.439100 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:31.439108 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:31.439279 (+ 171us) client_negotiation.cc:769] Sending connection context
0924 07:13:31.439433 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 07:13:31.439601 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
I20260924 07:13:31.440464 19721 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44379
I20260924 07:13:31.440554 19721 heartbeater.cc:464] Registering TS with master...
I20260924 07:13:31.440773 19721 heartbeater.cc:511] Master 127.0.98.62:44379 requested a full tablet report, sending...
I20260924 07:13:31.441273 19521 ts_manager.cc:194] Registered new tserver with Master: d924db38ce074bf080df04b7a84235d8 (127.0.98.1:45967)
I20260924 07:13:31.441861 19521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34145
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 07:13:31.463267 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44379
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:31.542243 19726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:31.542469 19726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:31.542505 19726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:31.543630 19726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:31.543695 19726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:31.543741 19726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:13:31.545224 19726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:31.545487 19726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:31.545720 19726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:31.550709 19731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:31.550690 19732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:31.550832 19726 server_base.cc:1034] running on GCE node
W20260924 07:13:31.550690 19734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:31.551028 19726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:31.551298 19726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:31.552454 19726 hybrid_clock.cc:648] HybridClock initialized: now 1790234011552442 us; error 45 us; skew 500 ppm
Sep 24 07:13:31 dist-test-slave-14ww krb5kdc[19485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:31.554785 19726 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 07:13:31.555204 19726 webserver.cc:460] Webserver started at http://127.0.98.2:39585/ using document root <none> and password file <none>
I20260924 07:13:31.555429 19726 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:31.555495 19726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:31.555639 19726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:31.556622 19726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-1/data/instance:
uuid: "0cd26e2adcd1489d91b6f73b916d0398"
format_stamp: "Formatted at 2026-09-24 07:13:31 on dist-test-slave-14ww"
server_key: "2df8eee87d4ef5a77c4e9dece7242e57"
server_key_iv: "05cdec857b6545c0085f2ec58c7a8e87"
server_key_version: "encryptionkey@0"
I20260924 07:13:31.557029 19726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0cd26e2adcd1489d91b6f73b916d0398"
format_stamp: "Formatted at 2026-09-24 07:13:31 on dist-test-slave-14ww"
server_key: "2df8eee87d4ef5a77c4e9dece7242e57"
server_key_iv: "05cdec857b6545c0085f2ec58c7a8e87"
server_key_version: "encryptionkey@0"
I20260924 07:13:31.558314 19726 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:13:31.559041 19741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:31.559206 19726 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:13:31.559297 19726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-1/wal
uuid: "0cd26e2adcd1489d91b6f73b916d0398"
format_stamp: "Formatted at 2026-09-24 07:13:31 on dist-test-slave-14ww"
server_key: "2df8eee87d4ef5a77c4e9dece7242e57"
server_key_iv: "05cdec857b6545c0085f2ec58c7a8e87"
server_key_version: "encryptionkey@0"
I20260924 07:13:31.559381 19726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:31.567876 19726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:31.569764 19726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:31.569919 19726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:31.570135 19726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:13:31.570418 19726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:13:31.570469 19726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:31.570503 19726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:13:31.570531 19726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:31.576129 19726 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43711
I20260924 07:13:31.576164 19854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43711 every 8 connection(s)
I20260924 07:13:31.576465 19726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:13:31.577394 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 19726
I20260924 07:13:31.577527 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:13:31.577729 392 external_mini_cluster.cc:1428] Setting key 07d2c4c25764df8d5664b7c6cd0e047d
Sep 24 07:13:31 dist-test-slave-14ww krb5kdc[19485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 07:13:31.587044 19570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:31.577237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58719 (local address 127.0.98.62:44379)
0924 07:13:31.577306 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:31.577308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:31.577523 (+ 215us) server_negotiation.cc:407] Connection header received
0924 07:13:31.578719 (+ 1196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:31.578720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:31.578733 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:31.578802 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:31.579372 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:31.580080 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:31.581652 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:31.581838 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:31.581888 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:31.584283 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:31.584284 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:31.584285 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:31.584292 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:31.585701 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:31.586085 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:31.586086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:31.586086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:31.586145 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:31.586412 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:31.586413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:31.586413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:31.586521 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:31.586602 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:31.586871 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 07:13:31.586955 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 07:13:31.587359 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:31.577220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44379 (local address 127.0.98.2:58719)
0924 07:13:31.577402 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:31.577441 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:31.578624 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:31.578853 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:31.578857 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:31.579103 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:31.579284 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:31.579288 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:31.580168 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:31.580169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:31.581560 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:31.581562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:31.581916 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:31.581918 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:31.582075 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:31.582578 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:31.582588 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:31.584186 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:31.585784 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:31.585790 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:31.585792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:31.586005 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:31.586211 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:31.586212 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:31.586213 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:31.586351 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:31.586603 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:31.586610 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:31.586744 (+ 134us) client_negotiation.cc:769] Sending connection context
0924 07:13:31.586850 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 07:13:31.586997 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":113,"threads_started":1}
I20260924 07:13:31.587795 19855 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44379
I20260924 07:13:31.587888 19855 heartbeater.cc:464] Registering TS with master...
I20260924 07:13:31.588021 19855 heartbeater.cc:511] Master 127.0.98.62:44379 requested a full tablet report, sending...
I20260924 07:13:31.588392 19521 ts_manager.cc:194] Registered new tserver with Master: 0cd26e2adcd1489d91b6f73b916d0398 (127.0.98.2:43711)
I20260924 07:13:31.588724 19521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58719
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 07:13:31.603837 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44379
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:31.683107 19860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:31.683300 19860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:31.683324 19860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:31.684406 19860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:31.684440 19860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:31.684466 19860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:13:31.685873 19860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:31.686056 19860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:31.686223 19860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:31.691601 19866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:31.691629 19865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:31.691670 19868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:31.691748 19860 server_base.cc:1034] running on GCE node
I20260924 07:13:31.691987 19860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:31.692225 19860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:31.693395 19860 hybrid_clock.cc:648] HybridClock initialized: now 1790234011693380 us; error 42 us; skew 500 ppm
Sep 24 07:13:31 dist-test-slave-14ww krb5kdc[19485](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:31.695868 19860 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 07:13:31.696352 19860 webserver.cc:460] Webserver started at http://127.0.98.3:43069/ using document root <none> and password file <none>
I20260924 07:13:31.696612 19860 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:31.696678 19860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:31.696826 19860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:31.697877 19860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-2/data/instance:
uuid: "8b90d1aa31a740b091aed15ff043af53"
format_stamp: "Formatted at 2026-09-24 07:13:31 on dist-test-slave-14ww"
server_key: "439e569633247d497f49bb9ad0c69fab"
server_key_iv: "fcf6d1cf3bede84f6fb04e7cc7c5ef77"
server_key_version: "encryptionkey@0"
I20260924 07:13:31.698244 19860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8b90d1aa31a740b091aed15ff043af53"
format_stamp: "Formatted at 2026-09-24 07:13:31 on dist-test-slave-14ww"
server_key: "439e569633247d497f49bb9ad0c69fab"
server_key_iv: "fcf6d1cf3bede84f6fb04e7cc7c5ef77"
server_key_version: "encryptionkey@0"
I20260924 07:13:31.699589 19860 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:13:31.700304 19875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:31.700542 19860 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:13:31.700657 19860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-2/wal
uuid: "8b90d1aa31a740b091aed15ff043af53"
format_stamp: "Formatted at 2026-09-24 07:13:31 on dist-test-slave-14ww"
server_key: "439e569633247d497f49bb9ad0c69fab"
server_key_iv: "fcf6d1cf3bede84f6fb04e7cc7c5ef77"
server_key_version: "encryptionkey@0"
I20260924 07:13:31.700768 19860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:31.710237 19860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:31.712059 19860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:31.712273 19860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:31.712540 19860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:13:31.712909 19860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:13:31.712975 19860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:31.713016 19860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:13:31.713044 19860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:31.719231 19860 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38275
I20260924 07:13:31.719262 19988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38275 every 8 connection(s)
I20260924 07:13:31.719652 19860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 07:13:31 dist-test-slave-14ww krb5kdc[19485](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234011, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:13:31.728222 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 19860
I20260924 07:13:31.728364 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:13:31.728523 19570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:31.720342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60717 (local address 127.0.98.62:44379)
0924 07:13:31.720407 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:31.720410 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:31.720614 (+ 204us) server_negotiation.cc:407] Connection header received
0924 07:13:31.721557 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:31.721559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:31.721571 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:31.721633 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:31.722161 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:31.722688 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:31.723637 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:31.723835 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:31.723893 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:31.725843 (+ 1950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:31.725845 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:31.725845 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:31.725852 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:31.727110 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:31.727524 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:31.727525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:31.727526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:31.727591 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:31.727832 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:31.727833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:31.727833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:31.727979 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:31.728092 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:31.728334 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 07:13:31.728425 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 07:13:31.728590 392 external_mini_cluster.cc:1428] Setting key 69b47cbc190e5763556391b0faecb581
I20260924 07:13:31.728670 19991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:31.720328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44379 (local address 127.0.98.3:60717)
0924 07:13:31.720491 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:31.720528 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:31.721456 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:31.721679 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:31.721682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:31.721909 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:31.722085 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:31.722089 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:31.722765 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:31.722766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:31.723545 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:31.723546 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:31.723909 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:31.723910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:31.724004 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:31.724407 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:31.724414 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:31.725760 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:31.727195 (+ 1435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:31.727201 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:31.727202 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:31.727419 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:31.727662 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:31.727662 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:31.727663 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:31.727752 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:31.728091 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:31.728095 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:31.728197 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 07:13:31.728276 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 07:13:31.728454 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
I20260924 07:13:31.729228 19989 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44379
I20260924 07:13:31.729326 19989 heartbeater.cc:464] Registering TS with master...
I20260924 07:13:31.729506 19989 heartbeater.cc:511] Master 127.0.98.62:44379 requested a full tablet report, sending...
I20260924 07:13:31.729831 19521 ts_manager.cc:194] Registered new tserver with Master: 8b90d1aa31a740b091aed15ff043af53 (127.0.98.3:38275)
I20260924 07:13:31.730207 19521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60717
I20260924 07:13:31.740711 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:13:31.748209 19570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:31.742254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60910 (local address 127.0.98.62:44379)
0924 07:13:31.742331 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:31.742333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:31.742393 (+ 60us) server_negotiation.cc:407] Connection header received
0924 07:13:31.742454 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:31.742455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:31.742464 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:31.742520 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:31.743101 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:31.743593 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:31.744344 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:31.744543 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:31.744601 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:31.745794 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:31.745795 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:31.745796 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:31.745801 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:31.747300 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:31.747595 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:31.747595 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:31.747596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:31.747644 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:31.747780 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:31.747780 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:31.747781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:31.747901 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:31.747968 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:31.748051 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 07:13:31.748103 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 07:13:31.750010 19521 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60910:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:13:31.751608 19521 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:13:31.758471 20001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:31.754103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47450 (local address 127.0.98.2:43711)
0924 07:13:31.754266 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:31.754271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:31.754567 (+ 296us) server_negotiation.cc:407] Connection header received
0924 07:13:31.754613 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:31.754616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:31.754631 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:31.754714 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:31.755057 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:31.756113 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:31.757302 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:31.758039 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:31.758121 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:31.758185 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:31.758337 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 07:13:31.758388 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":81,"threads_started":1}
I20260924 07:13:31.758466 19997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:31.753876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43711 (local address 127.0.0.1:47450)
0924 07:13:31.754452 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:31.754461 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:31.754523 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:31.754894 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:31.754895 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:31.754896 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:31.754936 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:31.754938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:31.756215 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:31.756216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:31.757212 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:31.757215 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:31.758133 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:31.758134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:31.758217 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:31.758231 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 07:13:31.758315 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 07:13:31.758377 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":52,"threads_started":1}
I20260924 07:13:31.758575 19998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:31.753818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45967 (local address 127.0.0.1:54396)
0924 07:13:31.754130 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:31.754174 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:31.754241 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:31.754744 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:31.754747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:31.754750 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:31.754818 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:31.754820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:31.755789 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:31.755790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:31.757271 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:31.757275 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:31.758224 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:31.758225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:31.758313 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:31.758323 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 07:13:31.758375 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 07:13:31.758421 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":6,"thread_start_us":60,"threads_started":1}
I20260924 07:13:31.758956 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:31.753922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54396 (local address 127.0.98.1:45967)
0924 07:13:31.754551 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:31.754556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:31.754566 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:13:31.754605 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:31.754607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:31.754620 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:31.754741 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:31.754879 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:31.755688 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:31.757354 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:31.758084 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:31.758141 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:31.758189 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:31.758510 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 07:13:31.758873 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":101,"threads_started":1}
I20260924 07:13:31.759317 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:31.753872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38275 (local address 127.0.0.1:52296)
0924 07:13:31.754343 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:31.754353 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:31.754390 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:31.754861 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:31.754862 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:31.754864 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:31.754912 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:31.754914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:31.756194 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:31.756195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:31.757628 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:31.757630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:31.758976 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:31.758977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:31.759125 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:31.759130 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 07:13:31.759199 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 07:13:31.759243 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":3,"thread_start_us":63,"threads_started":1}
I20260924 07:13:31.759348 20002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:31.754290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52296 (local address 127.0.98.3:38275)
0924 07:13:31.754441 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:31.754444 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:31.754452 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:13:31.754483 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:31.754485 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:31.754529 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:31.754869 (+ 340us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:31.754976 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:31.756077 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:31.757710 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:31.758257 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:31.758323 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:31.758372 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:31.759223 (+ 851us) server_negotiation.cc:299] Negotiation successful
0924 07:13:31.759267 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
I20260924 07:13:31.759398 19789 tablet_service.cc:1467] Processing CreateTablet for tablet 5d93d7a033a6409aafd61fb0cfe28aee (DEFAULT_TABLE table=test-table [id=bb9cb7e04cd94b2498faa669f88bf659]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:13:31.759775 19789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d93d7a033a6409aafd61fb0cfe28aee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:31.759686 19655 tablet_service.cc:1467] Processing CreateTablet for tablet 5d93d7a033a6409aafd61fb0cfe28aee (DEFAULT_TABLE table=test-table [id=bb9cb7e04cd94b2498faa669f88bf659]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:13:31.759987 19655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d93d7a033a6409aafd61fb0cfe28aee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:31.761456 19923 tablet_service.cc:1467] Processing CreateTablet for tablet 5d93d7a033a6409aafd61fb0cfe28aee (DEFAULT_TABLE table=test-table [id=bb9cb7e04cd94b2498faa669f88bf659]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:13:31.761790 19923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d93d7a033a6409aafd61fb0cfe28aee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:31.762833 20003 tablet_bootstrap.cc:492] T 5d93d7a033a6409aafd61fb0cfe28aee P 0cd26e2adcd1489d91b6f73b916d0398: Bootstrap starting.
I20260924 07:13:31.763036 20004 tablet_bootstrap.cc:492] T 5d93d7a033a6409aafd61fb0cfe28aee P d924db38ce074bf080df04b7a84235d8: Bootstrap starting.
I20260924 07:13:31.763582 20003 tablet_bootstrap.cc:654] T 5d93d7a033a6409aafd61fb0cfe28aee P 0cd26e2adcd1489d91b6f73b916d0398: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:31.763614 20004 tablet_bootstrap.cc:654] T 5d93d7a033a6409aafd61fb0cfe28aee P d924db38ce074bf080df04b7a84235d8: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:31.764187 20003 log.cc:824] T 5d93d7a033a6409aafd61fb0cfe28aee P 0cd26e2adcd1489d91b6f73b916d0398: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:31.764187 20004 log.cc:824] T 5d93d7a033a6409aafd61fb0cfe28aee P d924db38ce074bf080df04b7a84235d8: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:31.765112 20004 tablet_bootstrap.cc:492] T 5d93d7a033a6409aafd61fb0cfe28aee P d924db38ce074bf080df04b7a84235d8: No bootstrap required, opened a new log
I20260924 07:13:31.765168 20004 ts_tablet_manager.cc:1397] T 5d93d7a033a6409aafd61fb0cfe28aee P d924db38ce074bf080df04b7a84235d8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:13:31.765172 20003 tablet_bootstrap.cc:492] T 5d93d7a033a6409aafd61fb0cfe28aee P 0cd26e2adcd1489d91b6f73b916d0398: No bootstrap required, opened a new log
I20260924 07:13:31.765321 20003 ts_tablet_manager.cc:1397] T 5d93d7a033a6409aafd61fb0cfe28aee P 0cd26e2adcd1489d91b6f73b916d0398: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 07:13:31.766557 20004 raft_consensus.cc:348] T 5d93d7a033a6409aafd61fb0cfe28aee P d924db38ce074bf080df04b7a84235d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d924db38ce074bf080df04b7a84235d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 } } peers { permanent_uuid: "8b90d1aa31a740b091aed15ff043af53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38275 } } peers { permanent_uuid: "0cd26e2adcd1489d91b6f73b916d0398" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43711 } }
I20260924 07:13:31.766692 20004 raft_consensus.cc:374] T 5d93d7a033a6409aafd61fb0cfe28aee P d924db38ce074bf080df04b7a84235d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:31.766716 20004 raft_consensus.cc:729] T 5d93d7a033a6409aafd61fb0cfe28aee P d924db38ce074bf080df04b7a84235d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d924db38ce074bf080df04b7a84235d8, State: Initialized, Role: FOLLOWER
I20260924 07:13:31.766626 20003 raft_consensus.cc:348] T 5d93d7a033a6409aafd61fb0cfe28aee P 0cd26e2adcd1489d91b6f73b916d0398 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d924db38ce074bf080df04b7a84235d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 } } peers { permanent_uuid: "8b90d1aa31a740b091aed15ff043af53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38275 } } peers { permanent_uuid: "0cd26e2adcd1489d91b6f73b916d0398" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43711 } }
I20260924 07:13:31.766798 20003 raft_consensus.cc:374] T 5d93d7a033a6409aafd61fb0cfe28aee P 0cd26e2adcd1489d91b6f73b916d0398 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:31.766840 20003 raft_consensus.cc:729] T 5d93d7a033a6409aafd61fb0cfe28aee P 0cd26e2adcd1489d91b6f73b916d0398 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cd26e2adcd1489d91b6f73b916d0398, State: Initialized, Role: FOLLOWER
I20260924 07:13:31.766924 20004 consensus_queue.cc:260] T 5d93d7a033a6409aafd61fb0cfe28aee P d924db38ce074bf080df04b7a84235d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d924db38ce074bf080df04b7a84235d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 } } peers { permanent_uuid: "8b90d1aa31a740b091aed15ff043af53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38275 } } peers { permanent_uuid: "0cd26e2adcd1489d91b6f73b916d0398" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43711 } }
I20260924 07:13:31.766947 20003 consensus_queue.cc:260] T 5d93d7a033a6409aafd61fb0cfe28aee P 0cd26e2adcd1489d91b6f73b916d0398 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d924db38ce074bf080df04b7a84235d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 } } peers { permanent_uuid: "8b90d1aa31a740b091aed15ff043af53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38275 } } peers { permanent_uuid: "0cd26e2adcd1489d91b6f73b916d0398" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43711 } }
I20260924 07:13:31.767248 20003 ts_tablet_manager.cc:1428] T 5d93d7a033a6409aafd61fb0cfe28aee P 0cd26e2adcd1489d91b6f73b916d0398: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:13:31.767248 20004 ts_tablet_manager.cc:1428] T 5d93d7a033a6409aafd61fb0cfe28aee P d924db38ce074bf080df04b7a84235d8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:13:31.767252 19721 heartbeater.cc:503] Master 127.0.98.62:44379 was elected leader, sending a full tablet report...
I20260924 07:13:31.764612 20007 tablet_bootstrap.cc:492] T 5d93d7a033a6409aafd61fb0cfe28aee P 8b90d1aa31a740b091aed15ff043af53: Bootstrap starting.
I20260924 07:13:31.767403 19855 heartbeater.cc:503] Master 127.0.98.62:44379 was elected leader, sending a full tablet report...
I20260924 07:13:31.768107 20007 tablet_bootstrap.cc:654] T 5d93d7a033a6409aafd61fb0cfe28aee P 8b90d1aa31a740b091aed15ff043af53: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:31.768590 20007 log.cc:824] T 5d93d7a033a6409aafd61fb0cfe28aee P 8b90d1aa31a740b091aed15ff043af53: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:31.769467 20007 tablet_bootstrap.cc:492] T 5d93d7a033a6409aafd61fb0cfe28aee P 8b90d1aa31a740b091aed15ff043af53: No bootstrap required, opened a new log
I20260924 07:13:31.769541 20007 ts_tablet_manager.cc:1397] T 5d93d7a033a6409aafd61fb0cfe28aee P 8b90d1aa31a740b091aed15ff043af53: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 07:13:31.770884 20007 raft_consensus.cc:348] T 5d93d7a033a6409aafd61fb0cfe28aee P 8b90d1aa31a740b091aed15ff043af53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d924db38ce074bf080df04b7a84235d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 } } peers { permanent_uuid: "8b90d1aa31a740b091aed15ff043af53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38275 } } peers { permanent_uuid: "0cd26e2adcd1489d91b6f73b916d0398" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43711 } }
I20260924 07:13:31.771044 20007 raft_consensus.cc:374] T 5d93d7a033a6409aafd61fb0cfe28aee P 8b90d1aa31a740b091aed15ff043af53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:31.771098 20007 raft_consensus.cc:729] T 5d93d7a033a6409aafd61fb0cfe28aee P 8b90d1aa31a740b091aed15ff043af53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b90d1aa31a740b091aed15ff043af53, State: Initialized, Role: FOLLOWER
I20260924 07:13:31.771214 20007 consensus_queue.cc:260] T 5d93d7a033a6409aafd61fb0cfe28aee P 8b90d1aa31a740b091aed15ff043af53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d924db38ce074bf080df04b7a84235d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 } } peers { permanent_uuid: "8b90d1aa31a740b091aed15ff043af53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38275 } } peers { permanent_uuid: "0cd26e2adcd1489d91b6f73b916d0398" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43711 } }
I20260924 07:13:31.771492 20007 ts_tablet_manager.cc:1428] T 5d93d7a033a6409aafd61fb0cfe28aee P 8b90d1aa31a740b091aed15ff043af53: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:13:31.771566 19989 heartbeater.cc:503] Master 127.0.98.62:44379 was elected leader, sending a full tablet report...
I20260924 07:13:31.820317 20009 raft_consensus.cc:482] T 5d93d7a033a6409aafd61fb0cfe28aee P 0cd26e2adcd1489d91b6f73b916d0398 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:13:31.820446 20009 raft_consensus.cc:504] T 5d93d7a033a6409aafd61fb0cfe28aee P 0cd26e2adcd1489d91b6f73b916d0398 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d924db38ce074bf080df04b7a84235d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 } } peers { permanent_uuid: "8b90d1aa31a740b091aed15ff043af53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38275 } } peers { permanent_uuid: "0cd26e2adcd1489d91b6f73b916d0398" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43711 } }
I20260924 07:13:31.820847 20009 leader_election.cc:288] T 5d93d7a033a6409aafd61fb0cfe28aee P 0cd26e2adcd1489d91b6f73b916d0398 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d924db38ce074bf080df04b7a84235d8 (127.0.98.1:45967), 8b90d1aa31a740b091aed15ff043af53 (127.0.98.3:38275)
I20260924 07:13:31.823570 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:31.821010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45967 (local address 127.0.98.2:41989)
0924 07:13:31.821064 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:31.821077 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:31.821123 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:31.821297 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:31.821299 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:31.821301 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:31.821376 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:31.821377 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:31.822008 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:31.822009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:31.822744 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:31.822746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:31.823299 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:31.823300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:31.823383 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:31.823386 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:13:31.823460 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 07:13:31.823502 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 07:13:31.823567 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:31.821104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41989 (local address 127.0.98.1:45967)
0924 07:13:31.821145 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:31.821147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:31.821175 (+ 28us) server_negotiation.cc:407] Connection header received
0924 07:13:31.821204 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:31.821205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:31.821222 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:31.821290 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:31.821450 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:31.821922 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:31.822840 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:31.823216 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:31.823301 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:31.823311 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:31.823466 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 07:13:31.823502 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 07:13:31.823695 20002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:31.821085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37041 (local address 127.0.98.3:38275)
0924 07:13:31.821162 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:31.821164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:31.821294 (+ 130us) server_negotiation.cc:407] Connection header received
0924 07:13:31.821355 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:31.821356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:31.821366 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:31.821439 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:31.821589 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:31.822050 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:31.822953 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:31.823318 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:31.823398 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:31.823407 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:31.823572 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 07:13:31.823610 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 07:13:31.823868 20012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:31.821010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38275 (local address 127.0.98.2:37041)
0924 07:13:31.821226 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:31.821240 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:31.821300 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:31.821442 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:31.821443 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:31.821444 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:31.821519 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:31.821521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:31.822138 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:31.822139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:31.822866 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:31.822868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:31.823413 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:31.823415 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:31.823485 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:31.823488 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:13:31.823562 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 07:13:31.823599 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":49,"thread_start_us":112,"threads_started":1}
I20260924 07:13:31.823920 19675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d93d7a033a6409aafd61fb0cfe28aee" candidate_uuid: "0cd26e2adcd1489d91b6f73b916d0398" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d924db38ce074bf080df04b7a84235d8" is_pre_election: true
I20260924 07:13:31.824080 19675 raft_consensus.cc:2454] T 5d93d7a033a6409aafd61fb0cfe28aee P d924db38ce074bf080df04b7a84235d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cd26e2adcd1489d91b6f73b916d0398 in term 0.
I20260924 07:13:31.824241 19943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d93d7a033a6409aafd61fb0cfe28aee" candidate_uuid: "0cd26e2adcd1489d91b6f73b916d0398" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b90d1aa31a740b091aed15ff043af53" is_pre_election: true
I20260924 07:13:31.824318 19742 leader_election.cc:302] T 5d93d7a033a6409aafd61fb0cfe28aee P 0cd26e2adcd1489d91b6f73b916d0398 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cd26e2adcd1489d91b6f73b916d0398, d924db38ce074bf080df04b7a84235d8; no voters:
I20260924 07:13:31.824389 19943 raft_consensus.cc:2454] T 5d93d7a033a6409aafd61fb0cfe28aee P 8b90d1aa31a740b091aed15ff043af53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cd26e2adcd1489d91b6f73b916d0398 in term 0.
I20260924 07:13:31.824440 20009 raft_consensus.cc:2781] T 5d93d7a033a6409aafd61fb0cfe28aee P 0cd26e2adcd1489d91b6f73b916d0398 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:13:31.824515 20009 raft_consensus.cc:482] T 5d93d7a033a6409aafd61fb0cfe28aee P 0cd26e2adcd1489d91b6f73b916d0398 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:13:31.824540 20009 raft_consensus.cc:3037] T 5d93d7a033a6409aafd61fb0cfe28aee P 0cd26e2adcd1489d91b6f73b916d0398 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:31.825335 20009 raft_consensus.cc:504] T 5d93d7a033a6409aafd61fb0cfe28aee P 0cd26e2adcd1489d91b6f73b916d0398 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d924db38ce074bf080df04b7a84235d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 } } peers { permanent_uuid: "8b90d1aa31a740b091aed15ff043af53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38275 } } peers { permanent_uuid: "0cd26e2adcd1489d91b6f73b916d0398" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43711 } }
I20260924 07:13:31.825479 20009 leader_election.cc:288] T 5d93d7a033a6409aafd61fb0cfe28aee P 0cd26e2adcd1489d91b6f73b916d0398 [CANDIDATE]: Term 1 election: Requested vote from peers d924db38ce074bf080df04b7a84235d8 (127.0.98.1:45967), 8b90d1aa31a740b091aed15ff043af53 (127.0.98.3:38275)
I20260924 07:13:31.825618 19675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d93d7a033a6409aafd61fb0cfe28aee" candidate_uuid: "0cd26e2adcd1489d91b6f73b916d0398" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d924db38ce074bf080df04b7a84235d8"
I20260924 07:13:31.825675 19943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d93d7a033a6409aafd61fb0cfe28aee" candidate_uuid: "0cd26e2adcd1489d91b6f73b916d0398" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b90d1aa31a740b091aed15ff043af53"
I20260924 07:13:31.825695 19675 raft_consensus.cc:3037] T 5d93d7a033a6409aafd61fb0cfe28aee P d924db38ce074bf080df04b7a84235d8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:31.825773 19943 raft_consensus.cc:3037] T 5d93d7a033a6409aafd61fb0cfe28aee P 8b90d1aa31a740b091aed15ff043af53 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:31.826479 19675 raft_consensus.cc:2454] T 5d93d7a033a6409aafd61fb0cfe28aee P d924db38ce074bf080df04b7a84235d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cd26e2adcd1489d91b6f73b916d0398 in term 1.
I20260924 07:13:31.826575 19943 raft_consensus.cc:2454] T 5d93d7a033a6409aafd61fb0cfe28aee P 8b90d1aa31a740b091aed15ff043af53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cd26e2adcd1489d91b6f73b916d0398 in term 1.
I20260924 07:13:31.826618 19742 leader_election.cc:302] T 5d93d7a033a6409aafd61fb0cfe28aee P 0cd26e2adcd1489d91b6f73b916d0398 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cd26e2adcd1489d91b6f73b916d0398, d924db38ce074bf080df04b7a84235d8; no voters:
I20260924 07:13:31.826699 20009 raft_consensus.cc:2781] T 5d93d7a033a6409aafd61fb0cfe28aee P 0cd26e2adcd1489d91b6f73b916d0398 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:13:31.826844 20009 raft_consensus.cc:686] T 5d93d7a033a6409aafd61fb0cfe28aee P 0cd26e2adcd1489d91b6f73b916d0398 [term 1 LEADER]: Becoming Leader. State: Replica: 0cd26e2adcd1489d91b6f73b916d0398, State: Running, Role: LEADER
I20260924 07:13:31.826946 20009 consensus_queue.cc:237] T 5d93d7a033a6409aafd61fb0cfe28aee P 0cd26e2adcd1489d91b6f73b916d0398 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d924db38ce074bf080df04b7a84235d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 } } peers { permanent_uuid: "8b90d1aa31a740b091aed15ff043af53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38275 } } peers { permanent_uuid: "0cd26e2adcd1489d91b6f73b916d0398" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43711 } }
W20260924 07:13:31.826992 19856 tablet.cc:2369] T 5d93d7a033a6409aafd61fb0cfe28aee P 0cd26e2adcd1489d91b6f73b916d0398: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 07:13:31.827716 19521 catalog_manager.cc:5495] T 5d93d7a033a6409aafd61fb0cfe28aee P 0cd26e2adcd1489d91b6f73b916d0398 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cd26e2adcd1489d91b6f73b916d0398 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0cd26e2adcd1489d91b6f73b916d0398" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d924db38ce074bf080df04b7a84235d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b90d1aa31a740b091aed15ff043af53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cd26e2adcd1489d91b6f73b916d0398" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43711 } health_report { overall_health: HEALTHY } } }
I20260924 07:13:31.850044 20001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:31.847640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47464 (local address 127.0.98.2:43711)
0924 07:13:31.847704 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:31.847707 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:31.847715 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:13:31.847772 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:31.847773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:31.847790 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:31.847866 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:13:31.848051 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:31.848494 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:31.849267 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:31.849452 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:31.849526 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:31.849701 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:13:31.849796 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:13:31.849890 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:31.849913 (+ 23us) server_negotiation.cc:299] Negotiation successful
0924 07:13:31.849932 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 07:13:31.851783 19943 raft_consensus.cc:1261] T 5d93d7a033a6409aafd61fb0cfe28aee P 8b90d1aa31a740b091aed15ff043af53 [term 1 FOLLOWER]: Refusing update from remote peer 0cd26e2adcd1489d91b6f73b916d0398: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:13:31.851812 19675 raft_consensus.cc:1261] T 5d93d7a033a6409aafd61fb0cfe28aee P d924db38ce074bf080df04b7a84235d8 [term 1 FOLLOWER]: Refusing update from remote peer 0cd26e2adcd1489d91b6f73b916d0398: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:13:31.852186 20013 consensus_queue.cc:1035] T 5d93d7a033a6409aafd61fb0cfe28aee P 0cd26e2adcd1489d91b6f73b916d0398 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b90d1aa31a740b091aed15ff043af53" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:13:31.852326 20009 consensus_queue.cc:1035] T 5d93d7a033a6409aafd61fb0cfe28aee P 0cd26e2adcd1489d91b6f73b916d0398 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d924db38ce074bf080df04b7a84235d8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:13:31.855175 20017 mvcc.cc:205] Tried to move back new op lower bound from 7332798512542191616 to 7332798512443736064. Current Snapshot: MvccSnapshot[applied={T|T < 7332798512542191616}]
I20260924 07:13:31.855367 20018 mvcc.cc:205] Tried to move back new op lower bound from 7332798512542191616 to 7332798512443736064. Current Snapshot: MvccSnapshot[applied={T|T < 7332798512542191616}]
I20260924 07:13:31.856685 19530 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:13:31.856809 19530 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:13:31.857589 19530 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 72f253a9400f4868a2041067ef79f1c8: Sending DeleteTablet for 3 replicas of tablet 5d93d7a033a6409aafd61fb0cfe28aee
I20260924 07:13:31.857894 19923 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d93d7a033a6409aafd61fb0cfe28aee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:13:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52296
I20260924 07:13:31.857925 19789 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d93d7a033a6409aafd61fb0cfe28aee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:13:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47450
I20260924 07:13:31.858130 20026 tablet_replica.cc:330] T 5d93d7a033a6409aafd61fb0cfe28aee P 8b90d1aa31a740b091aed15ff043af53: stopping tablet replica
I20260924 07:13:31.858125 19655 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d93d7a033a6409aafd61fb0cfe28aee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:13:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54396
I20260924 07:13:31.858225 20026 raft_consensus.cc:2229] T 5d93d7a033a6409aafd61fb0cfe28aee P 8b90d1aa31a740b091aed15ff043af53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:13:31.858314 20026 raft_consensus.cc:2258] T 5d93d7a033a6409aafd61fb0cfe28aee P 8b90d1aa31a740b091aed15ff043af53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:13:31.858553 20028 tablet_replica.cc:330] T 5d93d7a033a6409aafd61fb0cfe28aee P d924db38ce074bf080df04b7a84235d8: stopping tablet replica
I20260924 07:13:31.858600 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 19592
I20260924 07:13:31.858654 20028 raft_consensus.cc:2229] T 5d93d7a033a6409aafd61fb0cfe28aee P d924db38ce074bf080df04b7a84235d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:13:31.859778 20027 tablet_replica.cc:330] T 5d93d7a033a6409aafd61fb0cfe28aee P 0cd26e2adcd1489d91b6f73b916d0398: stopping tablet replica
I20260924 07:13:31.859833 20026 ts_tablet_manager.cc:1905] T 5d93d7a033a6409aafd61fb0cfe28aee P 8b90d1aa31a740b091aed15ff043af53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:13:31.859959 20027 raft_consensus.cc:2229] T 5d93d7a033a6409aafd61fb0cfe28aee P 0cd26e2adcd1489d91b6f73b916d0398 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:13:31.860093 20027 raft_consensus.cc:2258] T 5d93d7a033a6409aafd61fb0cfe28aee P 0cd26e2adcd1489d91b6f73b916d0398 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:13:31.860496 20027 ts_tablet_manager.cc:1905] T 5d93d7a033a6409aafd61fb0cfe28aee P 0cd26e2adcd1489d91b6f73b916d0398: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:13:31.861810 20026 ts_tablet_manager.cc:1918] T 5d93d7a033a6409aafd61fb0cfe28aee P 8b90d1aa31a740b091aed15ff043af53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:13:31.861886 20026 log.cc:1196] T 5d93d7a033a6409aafd61fb0cfe28aee P 8b90d1aa31a740b091aed15ff043af53: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-2/wal/wals/5d93d7a033a6409aafd61fb0cfe28aee
I20260924 07:13:31.862174 20026 ts_tablet_manager.cc:1939] T 5d93d7a033a6409aafd61fb0cfe28aee P 8b90d1aa31a740b091aed15ff043af53: Deleting consensus metadata
I20260924 07:13:31.862583 19516 catalog_manager.cc:4841] TS 8b90d1aa31a740b091aed15ff043af53 (127.0.98.3:38275): tablet 5d93d7a033a6409aafd61fb0cfe28aee (table test-table [id=bb9cb7e04cd94b2498faa669f88bf659]) successfully deleted
I20260924 07:13:31.862900 20027 ts_tablet_manager.cc:1918] T 5d93d7a033a6409aafd61fb0cfe28aee P 0cd26e2adcd1489d91b6f73b916d0398: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:13:31.863027 20027 log.cc:1196] T 5d93d7a033a6409aafd61fb0cfe28aee P 0cd26e2adcd1489d91b6f73b916d0398: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790233772407728-392-0/minicluster-data/ts-1/wal/wals/5d93d7a033a6409aafd61fb0cfe28aee
I20260924 07:13:31.863377 20027 ts_tablet_manager.cc:1939] T 5d93d7a033a6409aafd61fb0cfe28aee P 0cd26e2adcd1489d91b6f73b916d0398: Deleting consensus metadata
I20260924 07:13:31.863777 19515 catalog_manager.cc:4841] TS 0cd26e2adcd1489d91b6f73b916d0398 (127.0.98.2:43711): tablet 5d93d7a033a6409aafd61fb0cfe28aee (table test-table [id=bb9cb7e04cd94b2498faa669f88bf659]) successfully deleted
W20260924 07:13:31.865065 19514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45967 (error 108)
I20260924 07:13:31.865556 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:31.865440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45967 (local address 127.0.0.1:54408)
0924 07:13:31.865486 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:31.865520 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45967: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 07:13:31.865682 19514 catalog_manager.cc:4584] TS d924db38ce074bf080df04b7a84235d8 (127.0.98.1:45967): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5d93d7a033a6409aafd61fb0cfe28aee: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45967: connect: Connection refused (error 111)
I20260924 07:13:31.866451 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 19726
I20260924 07:13:31.871326 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 19860
I20260924 07:13:31.875903 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 19498
2026-09-24T07:13:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1912 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T07:13:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:13:31Z Disabled control of system clock
I20260924 07:13:31.903364 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42357
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42357
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 07:13:31.992043 20033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:31.992233 20033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:31.992252 20033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:31.993503 20033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:13:31.993543 20033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:31.993556 20033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:31.993569 20033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:13:31.993582 20033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:13:31.994946 20033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42357
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:31.995173 20033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:31.995450 20033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:32.000149 20041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:32.000161 20039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:32.000180 20033 server_base.cc:1034] running on GCE node
W20260924 07:13:32.000150 20038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:32.000635 20033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:32.000942 20033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:32.002132 20033 hybrid_clock.cc:648] HybridClock initialized: now 1790234012002119 us; error 30 us; skew 500 ppm
I20260924 07:13:32.013428 20033 webserver.cc:460] Webserver started at http://127.0.98.62:45263/ using document root <none> and password file <none>
I20260924 07:13:32.013693 20033 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:32.013743 20033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:32.013922 20033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:32.014961 20033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "2fba9ae947db4de7bb0d7c16aa079d73"
format_stamp: "Formatted at 2026-09-24 07:13:32 on dist-test-slave-14ww"
server_key: "e513b530441491040336a50a042cb1a5"
server_key_iv: "9afd93ae5c064e5e0edcd72117fef9f3"
server_key_version: "encryptionkey@0"
I20260924 07:13:32.015321 20033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "2fba9ae947db4de7bb0d7c16aa079d73"
format_stamp: "Formatted at 2026-09-24 07:13:32 on dist-test-slave-14ww"
server_key: "e513b530441491040336a50a042cb1a5"
server_key_iv: "9afd93ae5c064e5e0edcd72117fef9f3"
server_key_version: "encryptionkey@0"
I20260924 07:13:32.016646 20033 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:13:32.017552 20047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:32.017773 20033 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:13:32.017841 20033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "2fba9ae947db4de7bb0d7c16aa079d73"
format_stamp: "Formatted at 2026-09-24 07:13:32 on dist-test-slave-14ww"
server_key: "e513b530441491040336a50a042cb1a5"
server_key_iv: "9afd93ae5c064e5e0edcd72117fef9f3"
server_key_version: "encryptionkey@0"
I20260924 07:13:32.017935 20033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:32.042716 20033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:32.043002 20033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:32.043146 20033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:32.046947 20033 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42357
I20260924 07:13:32.047004 20099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42357 every 8 connection(s)
I20260924 07:13:32.047310 20033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:13:32.047588 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 20033
I20260924 07:13:32.047706 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:13:32.047901 392 external_mini_cluster.cc:1428] Setting key cf399f1a6e3ebb2e291c8f202e069b8f
I20260924 07:13:32.048530 20100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:32.051626 20100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fba9ae947db4de7bb0d7c16aa079d73: Bootstrap starting.
I20260924 07:13:32.052342 20100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fba9ae947db4de7bb0d7c16aa079d73: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:32.052660 20100 log.cc:824] T 00000000000000000000000000000000 P 2fba9ae947db4de7bb0d7c16aa079d73: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:32.053555 20100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fba9ae947db4de7bb0d7c16aa079d73: No bootstrap required, opened a new log
I20260924 07:13:32.054136 20103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:32.048422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57004 (local address 127.0.98.62:42357)
0924 07:13:32.048624 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:32.048642 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:32.048674 (+ 32us) server_negotiation.cc:407] Connection header received
0924 07:13:32.049255 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:32.049264 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:32.049342 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:32.049493 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:32.049688 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:32.050998 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:32.051799 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:32.052403 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:32.052516 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:32.053432 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:32.053437 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:32.053437 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:13:32.053634 (+ 197us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:32.053674 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:13:32.053689 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:32.053761 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:32.053840 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 07:13:32.053921 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
I20260924 07:13:32.055804 20100 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fba9ae947db4de7bb0d7c16aa079d73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fba9ae947db4de7bb0d7c16aa079d73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42357 } }
I20260924 07:13:32.055985 20100 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fba9ae947db4de7bb0d7c16aa079d73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:32.056037 20100 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fba9ae947db4de7bb0d7c16aa079d73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fba9ae947db4de7bb0d7c16aa079d73, State: Initialized, Role: FOLLOWER
I20260924 07:13:32.056152 20100 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fba9ae947db4de7bb0d7c16aa079d73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fba9ae947db4de7bb0d7c16aa079d73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42357 } }
I20260924 07:13:32.056246 20100 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fba9ae947db4de7bb0d7c16aa079d73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:13:32.056288 20100 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fba9ae947db4de7bb0d7c16aa079d73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:13:32.056335 20100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fba9ae947db4de7bb0d7c16aa079d73 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:32.056933 20100 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fba9ae947db4de7bb0d7c16aa079d73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fba9ae947db4de7bb0d7c16aa079d73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42357 } }
I20260924 07:13:32.057049 20100 leader_election.cc:302] T 00000000000000000000000000000000 P 2fba9ae947db4de7bb0d7c16aa079d73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fba9ae947db4de7bb0d7c16aa079d73; no voters:
I20260924 07:13:32.057224 20100 leader_election.cc:288] T 00000000000000000000000000000000 P 2fba9ae947db4de7bb0d7c16aa079d73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:13:32.057274 20105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fba9ae947db4de7bb0d7c16aa079d73 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:13:32.057428 20105 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fba9ae947db4de7bb0d7c16aa079d73 [term 1 LEADER]: Becoming Leader. State: Replica: 2fba9ae947db4de7bb0d7c16aa079d73, State: Running, Role: LEADER
I20260924 07:13:32.057476 20100 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fba9ae947db4de7bb0d7c16aa079d73 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:13:32.057514 20105 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fba9ae947db4de7bb0d7c16aa079d73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fba9ae947db4de7bb0d7c16aa079d73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42357 } }
I20260924 07:13:32.057830 20106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fba9ae947db4de7bb0d7c16aa079d73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fba9ae947db4de7bb0d7c16aa079d73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fba9ae947db4de7bb0d7c16aa079d73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42357 } } }
I20260924 07:13:32.057842 20107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fba9ae947db4de7bb0d7c16aa079d73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fba9ae947db4de7bb0d7c16aa079d73. Latest consensus state: current_term: 1 leader_uuid: "2fba9ae947db4de7bb0d7c16aa079d73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fba9ae947db4de7bb0d7c16aa079d73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42357 } } }
I20260924 07:13:32.058022 20106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fba9ae947db4de7bb0d7c16aa079d73 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:32.058036 20107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fba9ae947db4de7bb0d7c16aa079d73 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:32.058426 20115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:13:32.058887 20115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:13:32.060325 20115 catalog_manager.cc:1347] Generated new cluster ID: 59fabc45969d4320ac398baaf26bc831
I20260924 07:13:32.060374 20115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:13:32.073318 20115 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:13:32.073961 20115 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:13:32.091926 20115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2fba9ae947db4de7bb0d7c16aa079d73: Generated new TSK 0
I20260924 07:13:32.092120 20115 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:13:32.097992 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42357
--builtin_ntp_servers=127.0.98.20:42699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 07:13:32.178428 20124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:32.178646 20124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:32.178682 20124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:32.179780 20124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:32.179839 20124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:32.179883 20124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 07:13:32.181277 20124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:32.181519 20124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:32.181799 20124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:32.187548 20132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:32.187553 20129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:32.187567 20130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:32.188230 20124 server_base.cc:1034] running on GCE node
I20260924 07:13:32.188431 20124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:32.188676 20124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:32.189855 20124 hybrid_clock.cc:648] HybridClock initialized: now 1790234012189840 us; error 27 us; skew 500 ppm
I20260924 07:13:32.191344 20124 webserver.cc:460] Webserver started at http://127.0.98.1:40157/ using document root <none> and password file <none>
I20260924 07:13:32.191604 20124 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:32.191660 20124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:32.191784 20124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:32.193123 20124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-0/data/instance:
uuid: "fff78e86fa25458dbda35eda724dea9c"
format_stamp: "Formatted at 2026-09-24 07:13:32 on dist-test-slave-14ww"
server_key: "a3b0d686c3a7664a411fa380090195ea"
server_key_iv: "7706dfd59790c291f0cec4f2850ed7d4"
server_key_version: "encryptionkey@0"
I20260924 07:13:32.193500 20124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fff78e86fa25458dbda35eda724dea9c"
format_stamp: "Formatted at 2026-09-24 07:13:32 on dist-test-slave-14ww"
server_key: "a3b0d686c3a7664a411fa380090195ea"
server_key_iv: "7706dfd59790c291f0cec4f2850ed7d4"
server_key_version: "encryptionkey@0"
I20260924 07:13:32.194978 20124 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:13:32.195760 20138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:32.196004 20124 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:13:32.196080 20124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-0/wal
uuid: "fff78e86fa25458dbda35eda724dea9c"
format_stamp: "Formatted at 2026-09-24 07:13:32 on dist-test-slave-14ww"
server_key: "a3b0d686c3a7664a411fa380090195ea"
server_key_iv: "7706dfd59790c291f0cec4f2850ed7d4"
server_key_version: "encryptionkey@0"
I20260924 07:13:32.196171 20124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:32.219300 20124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:32.219597 20124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:32.219746 20124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:32.219986 20124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:13:32.220356 20124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:13:32.220403 20124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:32.220449 20124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:13:32.220476 20124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:32.226409 20124 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38081
I20260924 07:13:32.226502 20251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38081 every 8 connection(s)
I20260924 07:13:32.226807 20124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-0/data/info.pb
I20260924 07:13:32.232270 20103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:32.227544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34023 (local address 127.0.98.62:42357)
0924 07:13:32.227611 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:32.227613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:32.228091 (+ 478us) server_negotiation.cc:407] Connection header received
0924 07:13:32.228636 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:32.228637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:32.228653 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:32.228701 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:32.229400 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:32.229897 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:32.230846 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:32.231052 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:32.231125 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:32.231966 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:32.231968 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:32.231969 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:13:32.231974 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:32.231990 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:13:32.231994 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:32.232065 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:32.232160 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 07:13:32.232188 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 07:13:32.232491 20254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:32.227535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42357 (local address 127.0.98.1:34023)
0924 07:13:32.227958 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:32.227993 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:32.228518 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:32.228798 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:32.228801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:32.228817 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:32.229293 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:32.229297 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:32.230006 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:32.230007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:32.230744 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:32.230745 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:32.231143 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:32.231144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:32.231268 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:32.231772 (+ 504us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:13:32.231773 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:32.231798 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:13:32.231820 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:32.232069 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:32.232075 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:32.232078 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:13:32.232161 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 07:13:32.232258 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":271,"threads_started":1}
I20260924 07:13:32.232759 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 20124
I20260924 07:13:32.232884 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-0/wal/instance
I20260924 07:13:32.233048 20252 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42357
I20260924 07:13:32.233135 20252 heartbeater.cc:464] Registering TS with master...
I20260924 07:13:32.233135 392 external_mini_cluster.cc:1428] Setting key 899afcace98d4c606b3589aa232bbfc0
I20260924 07:13:32.233309 20252 heartbeater.cc:511] Master 127.0.98.62:42357 requested a full tablet report, sending...
I20260924 07:13:32.233816 20063 ts_manager.cc:194] Registered new tserver with Master: fff78e86fa25458dbda35eda724dea9c (127.0.98.1:38081)
I20260924 07:13:32.234107 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42357
--builtin_ntp_servers=127.0.98.20:42699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 07:13:32.234540 20063 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:34023
W20260924 07:13:32.314277 20255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:32.314471 20255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:32.314491 20255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:32.315671 20255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:32.315711 20255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:32.315737 20255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 07:13:32.317112 20255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:32.317337 20255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:32.317617 20255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:32.322820 20260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:32.322829 20255 server_base.cc:1034] running on GCE node
W20260924 07:13:32.322844 20261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:32.323031 20263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:32.323282 20255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:32.323479 20255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:32.324625 20255 hybrid_clock.cc:648] HybridClock initialized: now 1790234012324612 us; error 28 us; skew 500 ppm
I20260924 07:13:32.325821 20255 webserver.cc:460] Webserver started at http://127.0.98.2:37113/ using document root <none> and password file <none>
I20260924 07:13:32.326052 20255 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:32.326133 20255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:32.326273 20255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:32.327348 20255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-1/data/instance:
uuid: "e181c4d0768544b4ba06893230783695"
format_stamp: "Formatted at 2026-09-24 07:13:32 on dist-test-slave-14ww"
server_key: "0d002acde54a6f3bcb5df4e9d1d52acc"
server_key_iv: "0ab5640b95d34cb64067b3a7b3518b9f"
server_key_version: "encryptionkey@0"
I20260924 07:13:32.327695 20255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e181c4d0768544b4ba06893230783695"
format_stamp: "Formatted at 2026-09-24 07:13:32 on dist-test-slave-14ww"
server_key: "0d002acde54a6f3bcb5df4e9d1d52acc"
server_key_iv: "0ab5640b95d34cb64067b3a7b3518b9f"
server_key_version: "encryptionkey@0"
I20260924 07:13:32.328958 20255 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:13:32.329669 20269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:32.329845 20255 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:32.329943 20255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-1/wal
uuid: "e181c4d0768544b4ba06893230783695"
format_stamp: "Formatted at 2026-09-24 07:13:32 on dist-test-slave-14ww"
server_key: "0d002acde54a6f3bcb5df4e9d1d52acc"
server_key_iv: "0ab5640b95d34cb64067b3a7b3518b9f"
server_key_version: "encryptionkey@0"
I20260924 07:13:32.330018 20255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:32.356541 20255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:32.356900 20255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:32.357064 20255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:32.357303 20255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:13:32.357648 20255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:13:32.357694 20255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:32.357735 20255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:13:32.357764 20255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:32.363540 20255 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34579
I20260924 07:13:32.363585 20382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34579 every 8 connection(s)
I20260924 07:13:32.363910 20255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-1/data/info.pb
I20260924 07:13:32.368854 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 20255
I20260924 07:13:32.368957 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-1/wal/instance
I20260924 07:13:32.369160 20103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:32.364620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52077 (local address 127.0.98.62:42357)
0924 07:13:32.364688 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:32.364690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:32.364940 (+ 250us) server_negotiation.cc:407] Connection header received
0924 07:13:32.365479 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:32.365481 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:32.365494 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:32.365557 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:32.366229 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:32.366729 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:32.367676 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:32.367893 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:32.367978 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:32.368843 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:32.368844 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:32.368844 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:13:32.368848 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:32.368862 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:13:32.368866 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:32.368939 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:32.369064 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 07:13:32.369085 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 07:13:32.369176 392 external_mini_cluster.cc:1428] Setting key 272a00e7cf604511e177dec3fbff00e6
I20260924 07:13:32.369403 20385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:32.364600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42357 (local address 127.0.98.2:52077)
0924 07:13:32.364829 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:32.364862 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:32.365358 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:32.365627 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:32.365631 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:32.365642 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:32.366138 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:32.366142 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:32.366821 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:32.366822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:32.367588 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:32.367590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:32.367988 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:32.367989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:32.368121 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:32.368630 (+ 509us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:13:32.368630 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:32.368682 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:13:32.368710 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:32.368945 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:32.368950 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:32.368954 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 07:13:32.369056 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 07:13:32.369195 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":100,"threads_started":1}
I20260924 07:13:32.369896 20383 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42357
I20260924 07:13:32.370010 20383 heartbeater.cc:464] Registering TS with master...
I20260924 07:13:32.370190 20383 heartbeater.cc:511] Master 127.0.98.62:42357 requested a full tablet report, sending...
I20260924 07:13:32.370399 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42357
--builtin_ntp_servers=127.0.98.20:42699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 07:13:32.370574 20063 ts_manager.cc:194] Registered new tserver with Master: e181c4d0768544b4ba06893230783695 (127.0.98.2:34579)
I20260924 07:13:32.371065 20063 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:52077
W20260924 07:13:32.449311 20386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:32.449494 20386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:32.449515 20386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:32.450558 20386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:32.450589 20386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:32.450613 20386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 07:13:32.451828 20386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:32.451987 20386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:32.452195 20386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:32.457059 20394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:32.457006 20392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:32.457023 20386 server_base.cc:1034] running on GCE node
W20260924 07:13:32.457000 20391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:32.457489 20386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:32.457710 20386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:32.458863 20386 hybrid_clock.cc:648] HybridClock initialized: now 1790234012458841 us; error 36 us; skew 500 ppm
I20260924 07:13:32.459941 20386 webserver.cc:460] Webserver started at http://127.0.98.3:44985/ using document root <none> and password file <none>
I20260924 07:13:32.460165 20386 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:32.460232 20386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:32.460361 20386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:32.461398 20386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-2/data/instance:
uuid: "4ad4c34b188a40309d8edfdfbd614552"
format_stamp: "Formatted at 2026-09-24 07:13:32 on dist-test-slave-14ww"
server_key: "e3ba67a6b2d9d2e7a8a84ac55b9aa568"
server_key_iv: "f78068d30dff529c7aea6d78b12c607e"
server_key_version: "encryptionkey@0"
I20260924 07:13:32.461755 20386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4ad4c34b188a40309d8edfdfbd614552"
format_stamp: "Formatted at 2026-09-24 07:13:32 on dist-test-slave-14ww"
server_key: "e3ba67a6b2d9d2e7a8a84ac55b9aa568"
server_key_iv: "f78068d30dff529c7aea6d78b12c607e"
server_key_version: "encryptionkey@0"
I20260924 07:13:32.462957 20386 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:13:32.463622 20400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:32.463785 20386 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:13:32.463878 20386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-2/wal
uuid: "4ad4c34b188a40309d8edfdfbd614552"
format_stamp: "Formatted at 2026-09-24 07:13:32 on dist-test-slave-14ww"
server_key: "e3ba67a6b2d9d2e7a8a84ac55b9aa568"
server_key_iv: "f78068d30dff529c7aea6d78b12c607e"
server_key_version: "encryptionkey@0"
I20260924 07:13:32.463953 20386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:32.475893 20386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:32.476154 20386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:32.476295 20386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:32.476505 20386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 07:13:32.476878 20386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 07:13:32.477025 20386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:32.477083 20386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 07:13:32.477113 20386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:32.483098 20386 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42933
I20260924 07:13:32.483127 20513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42933 every 8 connection(s)
I20260924 07:13:32.483443 20386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-2/data/info.pb
I20260924 07:13:32.484948 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 20386
I20260924 07:13:32.485064 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-2/wal/instance
I20260924 07:13:32.485262 392 external_mini_cluster.cc:1428] Setting key c9904d8c98f3f8cd828260ef71b08f42
I20260924 07:13:32.488818 20103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:32.484205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44485 (local address 127.0.98.62:42357)
0924 07:13:32.484283 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:32.484286 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:32.484471 (+ 185us) server_negotiation.cc:407] Connection header received
0924 07:13:32.485015 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:32.485016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:32.485029 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:32.485079 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:32.485920 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:32.486505 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:32.487467 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:32.487677 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:32.487751 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:32.488510 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:32.488511 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:32.488512 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:13:32.488516 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:32.488531 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:13:32.488535 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:32.488578 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:32.488651 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 07:13:32.488675 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 07:13:32.489001 20516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:32.484182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42357 (local address 127.0.98.3:44485)
0924 07:13:32.484364 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:32.484395 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:32.484900 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:32.485197 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:32.485202 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:32.485211 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:32.485848 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:32.485854 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:32.486598 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:32.486600 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:32.487368 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:32.487371 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:32.487770 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:32.487771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:32.487906 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:32.488361 (+ 455us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 07:13:32.488361 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:32.488386 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 07:13:32.488406 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:32.488589 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:32.488592 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:32.488595 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:13:32.488656 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 07:13:32.488776 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":102,"threads_started":1}
I20260924 07:13:32.489430 20514 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42357
I20260924 07:13:32.489530 20514 heartbeater.cc:464] Registering TS with master...
I20260924 07:13:32.489702 20514 heartbeater.cc:511] Master 127.0.98.62:42357 requested a full tablet report, sending...
I20260924 07:13:32.490077 20063 ts_manager.cc:194] Registered new tserver with Master: 4ad4c34b188a40309d8edfdfbd614552 (127.0.98.3:42933)
I20260924 07:13:32.490394 20063 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:44485
I20260924 07:13:32.496984 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 07:13:32.501215 20103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:32.498272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57006 (local address 127.0.98.62:42357)
0924 07:13:32.498339 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:32.498341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:32.498452 (+ 111us) server_negotiation.cc:407] Connection header received
0924 07:13:32.498520 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:32.498521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:32.498531 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:32.498585 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:32.498747 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:32.499231 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:32.500051 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:32.500225 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:32.500283 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:32.500960 (+ 677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:32.500961 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:32.500961 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:13:32.500964 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:32.500976 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:13:32.500980 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:32.501025 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:32.501090 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 07:13:32.501106 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 07:13:32.502611 20063 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 07:13:32.504225 20063 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 07:13:32.510613 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:32.506461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42933 (local address 127.0.0.1:40018)
0924 07:13:32.506873 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:32.506947 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:32.507007 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:32.507282 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:32.507285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:32.507288 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:32.507348 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:32.507350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:32.508807 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:32.508808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:32.509670 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:32.509672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:32.510332 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:32.510333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:32.510407 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:32.510420 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 07:13:32.510489 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 07:13:32.510539 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":2,"thread_start_us":63,"threads_started":1}
I20260924 07:13:32.510625 20525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:32.506520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40018 (local address 127.0.98.3:42933)
0924 07:13:32.506666 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:32.506670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:32.506994 (+ 324us) server_negotiation.cc:407] Connection header received
0924 07:13:32.507111 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:32.507113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:32.507128 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:32.507211 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:32.507438 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:32.508548 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:32.509754 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:32.510234 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:32.510327 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:32.510413 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:32.510506 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 07:13:32.510539 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":89,"threads_started":1}
I20260924 07:13:32.510982 20524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:32.506464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38081 (local address 127.0.0.1:58588)
0924 07:13:32.507090 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:32.507097 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:32.507158 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:32.507386 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:32.507387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:32.507390 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:32.507447 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:32.507449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:32.508649 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:32.508650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:32.509733 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:32.509735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:32.510708 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:32.510709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:32.510786 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:32.510792 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 07:13:32.510842 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 07:13:32.510880 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":5,"thread_start_us":90,"threads_started":1}
I20260924 07:13:32.511008 20526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:32.506691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45118 (local address 127.0.98.2:34579)
0924 07:13:32.506922 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:32.506925 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:32.507334 (+ 409us) server_negotiation.cc:407] Connection header received
0924 07:13:32.507378 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:32.507381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:32.507394 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:32.507464 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:32.507634 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:32.508306 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:32.509774 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:32.510521 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:32.510592 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:32.510670 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:32.510878 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 07:13:32.510918 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":91,"threads_started":1}
I20260924 07:13:32.510982 20522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:32.506491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34579 (local address 127.0.0.1:45118)
0924 07:13:32.507225 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:32.507232 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:32.507276 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:32.507527 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:32.507528 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:32.507529 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:32.507563 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:32.507563 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:32.508396 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:32.508397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:32.509674 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:32.509677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:32.510707 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:32.510708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:32.510786 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:32.510792 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 07:13:32.510849 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 07:13:32.510881 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"mutex_wait_us":66,"thread_start_us":66,"threads_started":1}
I20260924 07:13:32.511245 20527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:32.506689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58588 (local address 127.0.98.1:38081)
0924 07:13:32.506857 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:32.506860 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:32.507167 (+ 307us) server_negotiation.cc:407] Connection header received
0924 07:13:32.507226 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:32.507229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:32.507248 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:32.507317 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:32.507506 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:32.508568 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:32.509816 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:32.510529 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:32.510592 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:32.510670 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:32.511100 (+ 430us) server_negotiation.cc:299] Negotiation successful
0924 07:13:32.511146 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":97,"threads_started":1}
I20260924 07:13:32.511655 20448 tablet_service.cc:1467] Processing CreateTablet for tablet 708f3c1081214df0a40d4f2ea340790c (DEFAULT_TABLE table=test-table [id=a7a301575dd3470590ce9ca857b20c1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:13:32.512020 20448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 708f3c1081214df0a40d4f2ea340790c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:32.511988 20317 tablet_service.cc:1467] Processing CreateTablet for tablet 708f3c1081214df0a40d4f2ea340790c (DEFAULT_TABLE table=test-table [id=a7a301575dd3470590ce9ca857b20c1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:13:32.512283 20317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 708f3c1081214df0a40d4f2ea340790c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:32.514839 20528 tablet_bootstrap.cc:492] T 708f3c1081214df0a40d4f2ea340790c P 4ad4c34b188a40309d8edfdfbd614552: Bootstrap starting.
I20260924 07:13:32.515127 20186 tablet_service.cc:1467] Processing CreateTablet for tablet 708f3c1081214df0a40d4f2ea340790c (DEFAULT_TABLE table=test-table [id=a7a301575dd3470590ce9ca857b20c1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 07:13:32.515410 20186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 708f3c1081214df0a40d4f2ea340790c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:32.515451 20528 tablet_bootstrap.cc:654] T 708f3c1081214df0a40d4f2ea340790c P 4ad4c34b188a40309d8edfdfbd614552: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:32.515728 20528 log.cc:824] T 708f3c1081214df0a40d4f2ea340790c P 4ad4c34b188a40309d8edfdfbd614552: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:32.516351 20528 tablet_bootstrap.cc:492] T 708f3c1081214df0a40d4f2ea340790c P 4ad4c34b188a40309d8edfdfbd614552: No bootstrap required, opened a new log
I20260924 07:13:32.516402 20528 ts_tablet_manager.cc:1397] T 708f3c1081214df0a40d4f2ea340790c P 4ad4c34b188a40309d8edfdfbd614552: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:13:32.517984 20531 tablet_bootstrap.cc:492] T 708f3c1081214df0a40d4f2ea340790c P fff78e86fa25458dbda35eda724dea9c: Bootstrap starting.
I20260924 07:13:32.518222 20529 tablet_bootstrap.cc:492] T 708f3c1081214df0a40d4f2ea340790c P e181c4d0768544b4ba06893230783695: Bootstrap starting.
I20260924 07:13:32.518478 20528 raft_consensus.cc:348] T 708f3c1081214df0a40d4f2ea340790c P 4ad4c34b188a40309d8edfdfbd614552 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fff78e86fa25458dbda35eda724dea9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38081 } } peers { permanent_uuid: "4ad4c34b188a40309d8edfdfbd614552" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42933 } } peers { permanent_uuid: "e181c4d0768544b4ba06893230783695" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34579 } }
I20260924 07:13:32.518632 20528 raft_consensus.cc:374] T 708f3c1081214df0a40d4f2ea340790c P 4ad4c34b188a40309d8edfdfbd614552 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:32.518674 20528 raft_consensus.cc:729] T 708f3c1081214df0a40d4f2ea340790c P 4ad4c34b188a40309d8edfdfbd614552 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ad4c34b188a40309d8edfdfbd614552, State: Initialized, Role: FOLLOWER
I20260924 07:13:32.518690 20531 tablet_bootstrap.cc:654] T 708f3c1081214df0a40d4f2ea340790c P fff78e86fa25458dbda35eda724dea9c: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:32.518832 20529 tablet_bootstrap.cc:654] T 708f3c1081214df0a40d4f2ea340790c P e181c4d0768544b4ba06893230783695: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:32.518816 20528 consensus_queue.cc:260] T 708f3c1081214df0a40d4f2ea340790c P 4ad4c34b188a40309d8edfdfbd614552 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fff78e86fa25458dbda35eda724dea9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38081 } } peers { permanent_uuid: "4ad4c34b188a40309d8edfdfbd614552" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42933 } } peers { permanent_uuid: "e181c4d0768544b4ba06893230783695" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34579 } }
I20260924 07:13:32.518989 20531 log.cc:824] T 708f3c1081214df0a40d4f2ea340790c P fff78e86fa25458dbda35eda724dea9c: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:32.519147 20529 log.cc:824] T 708f3c1081214df0a40d4f2ea340790c P e181c4d0768544b4ba06893230783695: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:32.519284 20528 ts_tablet_manager.cc:1428] T 708f3c1081214df0a40d4f2ea340790c P 4ad4c34b188a40309d8edfdfbd614552: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 07:13:32.519421 20514 heartbeater.cc:503] Master 127.0.98.62:42357 was elected leader, sending a full tablet report...
I20260924 07:13:32.519877 20531 tablet_bootstrap.cc:492] T 708f3c1081214df0a40d4f2ea340790c P fff78e86fa25458dbda35eda724dea9c: No bootstrap required, opened a new log
I20260924 07:13:32.519899 20529 tablet_bootstrap.cc:492] T 708f3c1081214df0a40d4f2ea340790c P e181c4d0768544b4ba06893230783695: No bootstrap required, opened a new log
I20260924 07:13:32.519963 20531 ts_tablet_manager.cc:1397] T 708f3c1081214df0a40d4f2ea340790c P fff78e86fa25458dbda35eda724dea9c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 07:13:32.519994 20529 ts_tablet_manager.cc:1397] T 708f3c1081214df0a40d4f2ea340790c P e181c4d0768544b4ba06893230783695: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 07:13:32.522291 20531 raft_consensus.cc:348] T 708f3c1081214df0a40d4f2ea340790c P fff78e86fa25458dbda35eda724dea9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fff78e86fa25458dbda35eda724dea9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38081 } } peers { permanent_uuid: "4ad4c34b188a40309d8edfdfbd614552" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42933 } } peers { permanent_uuid: "e181c4d0768544b4ba06893230783695" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34579 } }
I20260924 07:13:32.522423 20531 raft_consensus.cc:374] T 708f3c1081214df0a40d4f2ea340790c P fff78e86fa25458dbda35eda724dea9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:32.522444 20531 raft_consensus.cc:729] T 708f3c1081214df0a40d4f2ea340790c P fff78e86fa25458dbda35eda724dea9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fff78e86fa25458dbda35eda724dea9c, State: Initialized, Role: FOLLOWER
I20260924 07:13:32.522360 20529 raft_consensus.cc:348] T 708f3c1081214df0a40d4f2ea340790c P e181c4d0768544b4ba06893230783695 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fff78e86fa25458dbda35eda724dea9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38081 } } peers { permanent_uuid: "4ad4c34b188a40309d8edfdfbd614552" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42933 } } peers { permanent_uuid: "e181c4d0768544b4ba06893230783695" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34579 } }
I20260924 07:13:32.522468 20529 raft_consensus.cc:374] T 708f3c1081214df0a40d4f2ea340790c P e181c4d0768544b4ba06893230783695 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:32.522498 20529 raft_consensus.cc:729] T 708f3c1081214df0a40d4f2ea340790c P e181c4d0768544b4ba06893230783695 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e181c4d0768544b4ba06893230783695, State: Initialized, Role: FOLLOWER
I20260924 07:13:32.522548 20531 consensus_queue.cc:260] T 708f3c1081214df0a40d4f2ea340790c P fff78e86fa25458dbda35eda724dea9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fff78e86fa25458dbda35eda724dea9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38081 } } peers { permanent_uuid: "4ad4c34b188a40309d8edfdfbd614552" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42933 } } peers { permanent_uuid: "e181c4d0768544b4ba06893230783695" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34579 } }
I20260924 07:13:32.522604 20529 consensus_queue.cc:260] T 708f3c1081214df0a40d4f2ea340790c P e181c4d0768544b4ba06893230783695 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fff78e86fa25458dbda35eda724dea9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38081 } } peers { permanent_uuid: "4ad4c34b188a40309d8edfdfbd614552" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42933 } } peers { permanent_uuid: "e181c4d0768544b4ba06893230783695" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34579 } }
I20260924 07:13:32.522756 20531 ts_tablet_manager.cc:1428] T 708f3c1081214df0a40d4f2ea340790c P fff78e86fa25458dbda35eda724dea9c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 07:13:32.522830 20252 heartbeater.cc:503] Master 127.0.98.62:42357 was elected leader, sending a full tablet report...
I20260924 07:13:32.522851 20529 ts_tablet_manager.cc:1428] T 708f3c1081214df0a40d4f2ea340790c P e181c4d0768544b4ba06893230783695: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 07:13:32.523088 20383 heartbeater.cc:503] Master 127.0.98.62:42357 was elected leader, sending a full tablet report...
I20260924 07:13:32.529048 20532 raft_consensus.cc:482] T 708f3c1081214df0a40d4f2ea340790c P 4ad4c34b188a40309d8edfdfbd614552 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 07:13:32.529148 20532 raft_consensus.cc:504] T 708f3c1081214df0a40d4f2ea340790c P 4ad4c34b188a40309d8edfdfbd614552 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fff78e86fa25458dbda35eda724dea9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38081 } } peers { permanent_uuid: "4ad4c34b188a40309d8edfdfbd614552" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42933 } } peers { permanent_uuid: "e181c4d0768544b4ba06893230783695" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34579 } }
I20260924 07:13:32.529466 20532 leader_election.cc:288] T 708f3c1081214df0a40d4f2ea340790c P 4ad4c34b188a40309d8edfdfbd614552 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fff78e86fa25458dbda35eda724dea9c (127.0.98.1:38081), e181c4d0768544b4ba06893230783695 (127.0.98.2:34579)
I20260924 07:13:32.532167 20526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:32.529747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52283 (local address 127.0.98.2:34579)
0924 07:13:32.529788 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:32.529790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:32.529801 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:13:32.529828 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:32.529829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:32.529838 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:32.529894 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:32.530017 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:32.530542 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:32.531456 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:32.531804 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:32.531911 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:32.531920 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:32.532068 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 07:13:32.532103 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 07:13:32.532168 20516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:32.529655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34579 (local address 127.0.98.3:52283)
0924 07:13:32.529720 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:32.529732 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:32.529778 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:32.529897 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:32.529898 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:32.529899 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:32.529963 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:32.529965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:32.530635 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:32.530636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:32.531355 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:32.531356 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:32.531910 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:32.531911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:32.531985 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:32.531988 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:13:32.532061 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 07:13:32.532105 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 07:13:32.532423 20537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:32.529700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38081 (local address 127.0.98.3:50907)
0924 07:13:32.529844 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:32.529853 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:32.529910 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:32.530099 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:32.530100 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:32.530102 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 07:13:32.530155 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:32.530156 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:32.530760 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:32.530761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:32.531495 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:32.531496 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:32.532122 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:32.532123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:32.532195 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:32.532198 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 07:13:32.532303 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 07:13:32.532352 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":88,"threads_started":1}
I20260924 07:13:32.532503 20527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:32.529952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50907 (local address 127.0.98.1:38081)
0924 07:13:32.530008 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:32.530010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:32.530016 (+ 6us) server_negotiation.cc:407] Connection header received
0924 07:13:32.530035 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:32.530036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:32.530043 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:32.530078 (+ 35us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 07:13:32.530212 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:32.530666 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:32.531581 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:32.532021 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:32.532109 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:32.532119 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:32.532389 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 07:13:32.532434 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 07:13:32.532471 20337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708f3c1081214df0a40d4f2ea340790c" candidate_uuid: "4ad4c34b188a40309d8edfdfbd614552" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e181c4d0768544b4ba06893230783695" is_pre_election: true
I20260924 07:13:32.532596 20337 raft_consensus.cc:2454] T 708f3c1081214df0a40d4f2ea340790c P e181c4d0768544b4ba06893230783695 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ad4c34b188a40309d8edfdfbd614552 in term 0.
I20260924 07:13:32.532824 20403 leader_election.cc:302] T 708f3c1081214df0a40d4f2ea340790c P 4ad4c34b188a40309d8edfdfbd614552 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ad4c34b188a40309d8edfdfbd614552, e181c4d0768544b4ba06893230783695; no voters:
I20260924 07:13:32.532840 20206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708f3c1081214df0a40d4f2ea340790c" candidate_uuid: "4ad4c34b188a40309d8edfdfbd614552" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fff78e86fa25458dbda35eda724dea9c" is_pre_election: true
I20260924 07:13:32.532953 20532 raft_consensus.cc:2781] T 708f3c1081214df0a40d4f2ea340790c P 4ad4c34b188a40309d8edfdfbd614552 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 07:13:32.532986 20206 raft_consensus.cc:2454] T 708f3c1081214df0a40d4f2ea340790c P fff78e86fa25458dbda35eda724dea9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ad4c34b188a40309d8edfdfbd614552 in term 0.
I20260924 07:13:32.533006 20532 raft_consensus.cc:482] T 708f3c1081214df0a40d4f2ea340790c P 4ad4c34b188a40309d8edfdfbd614552 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 07:13:32.533034 20532 raft_consensus.cc:3037] T 708f3c1081214df0a40d4f2ea340790c P 4ad4c34b188a40309d8edfdfbd614552 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:32.533787 20532 raft_consensus.cc:504] T 708f3c1081214df0a40d4f2ea340790c P 4ad4c34b188a40309d8edfdfbd614552 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fff78e86fa25458dbda35eda724dea9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38081 } } peers { permanent_uuid: "4ad4c34b188a40309d8edfdfbd614552" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42933 } } peers { permanent_uuid: "e181c4d0768544b4ba06893230783695" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34579 } }
I20260924 07:13:32.533929 20532 leader_election.cc:288] T 708f3c1081214df0a40d4f2ea340790c P 4ad4c34b188a40309d8edfdfbd614552 [CANDIDATE]: Term 1 election: Requested vote from peers fff78e86fa25458dbda35eda724dea9c (127.0.98.1:38081), e181c4d0768544b4ba06893230783695 (127.0.98.2:34579)
I20260924 07:13:32.534055 20206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708f3c1081214df0a40d4f2ea340790c" candidate_uuid: "4ad4c34b188a40309d8edfdfbd614552" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fff78e86fa25458dbda35eda724dea9c"
I20260924 07:13:32.534101 20337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708f3c1081214df0a40d4f2ea340790c" candidate_uuid: "4ad4c34b188a40309d8edfdfbd614552" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e181c4d0768544b4ba06893230783695"
I20260924 07:13:32.534152 20206 raft_consensus.cc:3037] T 708f3c1081214df0a40d4f2ea340790c P fff78e86fa25458dbda35eda724dea9c [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:32.534159 20337 raft_consensus.cc:3037] T 708f3c1081214df0a40d4f2ea340790c P e181c4d0768544b4ba06893230783695 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:32.534754 20337 raft_consensus.cc:2454] T 708f3c1081214df0a40d4f2ea340790c P e181c4d0768544b4ba06893230783695 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ad4c34b188a40309d8edfdfbd614552 in term 1.
I20260924 07:13:32.534868 20206 raft_consensus.cc:2454] T 708f3c1081214df0a40d4f2ea340790c P fff78e86fa25458dbda35eda724dea9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ad4c34b188a40309d8edfdfbd614552 in term 1.
I20260924 07:13:32.534900 20403 leader_election.cc:302] T 708f3c1081214df0a40d4f2ea340790c P 4ad4c34b188a40309d8edfdfbd614552 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ad4c34b188a40309d8edfdfbd614552, e181c4d0768544b4ba06893230783695; no voters:
I20260924 07:13:32.534984 20532 raft_consensus.cc:2781] T 708f3c1081214df0a40d4f2ea340790c P 4ad4c34b188a40309d8edfdfbd614552 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:13:32.535117 20532 raft_consensus.cc:686] T 708f3c1081214df0a40d4f2ea340790c P 4ad4c34b188a40309d8edfdfbd614552 [term 1 LEADER]: Becoming Leader. State: Replica: 4ad4c34b188a40309d8edfdfbd614552, State: Running, Role: LEADER
I20260924 07:13:32.535182 20532 consensus_queue.cc:237] T 708f3c1081214df0a40d4f2ea340790c P 4ad4c34b188a40309d8edfdfbd614552 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fff78e86fa25458dbda35eda724dea9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38081 } } peers { permanent_uuid: "4ad4c34b188a40309d8edfdfbd614552" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42933 } } peers { permanent_uuid: "e181c4d0768544b4ba06893230783695" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34579 } }
I20260924 07:13:32.535874 20063 catalog_manager.cc:5495] T 708f3c1081214df0a40d4f2ea340790c P 4ad4c34b188a40309d8edfdfbd614552 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ad4c34b188a40309d8edfdfbd614552 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "4ad4c34b188a40309d8edfdfbd614552" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fff78e86fa25458dbda35eda724dea9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ad4c34b188a40309d8edfdfbd614552" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e181c4d0768544b4ba06893230783695" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34579 } health_report { overall_health: UNKNOWN } } }
I20260924 07:13:32.545606 20525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:32.543502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40032 (local address 127.0.98.3:42933)
0924 07:13:32.543561 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:32.543563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:32.543570 (+ 7us) server_negotiation.cc:407] Connection header received
0924 07:13:32.543642 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:32.543642 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:32.543656 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:32.543711 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:13:32.543815 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:32.544244 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:32.544995 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:32.545164 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:32.545213 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:32.545327 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:13:32.545407 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:13:32.545462 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:32.545493 (+ 31us) server_negotiation.cc:299] Negotiation successful
0924 07:13:32.545505 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 07:13:32.547174 20337 raft_consensus.cc:1261] T 708f3c1081214df0a40d4f2ea340790c P e181c4d0768544b4ba06893230783695 [term 1 FOLLOWER]: Refusing update from remote peer 4ad4c34b188a40309d8edfdfbd614552: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:13:32.547286 20206 raft_consensus.cc:1261] T 708f3c1081214df0a40d4f2ea340790c P fff78e86fa25458dbda35eda724dea9c [term 1 FOLLOWER]: Refusing update from remote peer 4ad4c34b188a40309d8edfdfbd614552: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:13:32.547531 20538 consensus_queue.cc:1035] T 708f3c1081214df0a40d4f2ea340790c P 4ad4c34b188a40309d8edfdfbd614552 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e181c4d0768544b4ba06893230783695" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:13:32.547652 20532 consensus_queue.cc:1035] T 708f3c1081214df0a40d4f2ea340790c P 4ad4c34b188a40309d8edfdfbd614552 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fff78e86fa25458dbda35eda724dea9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:13:32.552469 20064 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:13:32.552546 20064 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 07:13:32.553310 20064 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2fba9ae947db4de7bb0d7c16aa079d73: Sending DeleteTablet for 3 replicas of tablet 708f3c1081214df0a40d4f2ea340790c
I20260924 07:13:32.553654 20186 tablet_service.cc:1514] Processing DeleteTablet for tablet 708f3c1081214df0a40d4f2ea340790c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:13:32 UTC) from {username='slave'} at 127.0.0.1:58588
I20260924 07:13:32.553674 20317 tablet_service.cc:1514] Processing DeleteTablet for tablet 708f3c1081214df0a40d4f2ea340790c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:13:32 UTC) from {username='slave'} at 127.0.0.1:45118
I20260924 07:13:32.553900 20551 tablet_replica.cc:330] T 708f3c1081214df0a40d4f2ea340790c P fff78e86fa25458dbda35eda724dea9c: stopping tablet replica
I20260924 07:13:32.553956 20448 tablet_service.cc:1514] Processing DeleteTablet for tablet 708f3c1081214df0a40d4f2ea340790c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 07:13:32 UTC) from {username='slave'} at 127.0.0.1:40018
I20260924 07:13:32.553992 20551 raft_consensus.cc:2229] T 708f3c1081214df0a40d4f2ea340790c P fff78e86fa25458dbda35eda724dea9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 07:13:32.554081 20551 raft_consensus.cc:2258] T 708f3c1081214df0a40d4f2ea340790c P fff78e86fa25458dbda35eda724dea9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:13:32.554318 20553 tablet_replica.cc:330] T 708f3c1081214df0a40d4f2ea340790c P 4ad4c34b188a40309d8edfdfbd614552: stopping tablet replica
I20260924 07:13:32.554373 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 20124
I20260924 07:13:32.554412 20553 raft_consensus.cc:2229] T 708f3c1081214df0a40d4f2ea340790c P 4ad4c34b188a40309d8edfdfbd614552 [term 1 LEADER]: Raft consensus shutting down.
I20260924 07:13:32.554495 20553 raft_consensus.cc:2258] T 708f3c1081214df0a40d4f2ea340790c P 4ad4c34b188a40309d8edfdfbd614552 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 07:13:32.555529 20553 ts_tablet_manager.cc:1905] T 708f3c1081214df0a40d4f2ea340790c P 4ad4c34b188a40309d8edfdfbd614552: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 07:13:32.557550 20553 ts_tablet_manager.cc:1918] T 708f3c1081214df0a40d4f2ea340790c P 4ad4c34b188a40309d8edfdfbd614552: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 07:13:32.557614 20553 log.cc:1196] T 708f3c1081214df0a40d4f2ea340790c P 4ad4c34b188a40309d8edfdfbd614552: Deleting WAL directory at /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790233772407728-392-0/minicluster-data/ts-2/wal/wals/708f3c1081214df0a40d4f2ea340790c
I20260924 07:13:32.557883 20553 ts_tablet_manager.cc:1939] T 708f3c1081214df0a40d4f2ea340790c P 4ad4c34b188a40309d8edfdfbd614552: Deleting consensus metadata
I20260924 07:13:32.558483 20048 catalog_manager.cc:4841] TS 4ad4c34b188a40309d8edfdfbd614552 (127.0.98.3:42933): tablet 708f3c1081214df0a40d4f2ea340790c (table test-table [id=a7a301575dd3470590ce9ca857b20c1f]) successfully deleted
W20260924 07:13:32.561425 20049 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38081 (error 108)
I20260924 07:13:32.561643 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 20255
I20260924 07:13:32.561900 20522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:32.561777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38081 (local address 127.0.0.1:58592)
0924 07:13:32.561821 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:32.561852 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 07:13:32.562002 20049 catalog_manager.cc:4584] TS fff78e86fa25458dbda35eda724dea9c (127.0.98.1:38081): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 708f3c1081214df0a40d4f2ea340790c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38081: connect: Connection refused (error 111)
I20260924 07:13:32.569605 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 20386
I20260924 07:13:32.569626 20522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:32.569533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34579 (local address 127.0.0.1:45126)
0924 07:13:32.569570 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:32.569603 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260924 07:13:32.574322 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 20033
2026-09-24T07:13:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (697 ms)
[----------] 6 tests from EncryptionPolicyTest (8326 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:13:32 dist-test-slave-14ww krb5kdc[20557](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:13:32 dist-test-slave-14ww krb5kdc[20557](info): set up 2 sockets
Sep 24 07:13:32 dist-test-slave-14ww krb5kdc[20557](info): commencing operation
krb5kdc: starting...
W20260924 07:13:33.612800 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:13:33 dist-test-slave-14ww krb5kdc[20557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:13:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:13:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:13:33.729417 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42551
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:33.807828 20570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:33.808051 20570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:33.808099 20570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:33.809247 20570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:13:33.809314 20570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:33.809337 20570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:33.809366 20570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:13:33.809398 20570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:13:33.810839 20570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42551
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:33.811110 20570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:33.811331 20570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:33.816093 20575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:33.816116 20578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:33.816116 20576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:33.816160 20570 server_base.cc:1034] running on GCE node
I20260924 07:13:33.816517 20570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:33.816805 20570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:33.817993 20570 hybrid_clock.cc:648] HybridClock initialized: now 1790234013817968 us; error 39 us; skew 500 ppm
Sep 24 07:13:33 dist-test-slave-14ww krb5kdc[20557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234013, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:33.820487 20570 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:13:33.820947 20570 webserver.cc:460] Webserver started at http://127.0.0.1:40059/ using document root <none> and password file <none>
I20260924 07:13:33.821125 20570 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:33.821164 20570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:33.821249 20570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:33.822216 20570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "dec504cb31a9433abfc3870c56fd46d3"
format_stamp: "Formatted at 2026-09-24 07:13:33 on dist-test-slave-14ww"
server_key: "6216c8857618f6e494549667860cf510"
server_key_iv: "962e6bd70d282da1b18941c9e72b327e"
server_key_version: "encryptionkey@0"
I20260924 07:13:33.822515 20570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "dec504cb31a9433abfc3870c56fd46d3"
format_stamp: "Formatted at 2026-09-24 07:13:33 on dist-test-slave-14ww"
server_key: "6216c8857618f6e494549667860cf510"
server_key_iv: "962e6bd70d282da1b18941c9e72b327e"
server_key_version: "encryptionkey@0"
I20260924 07:13:33.823734 20570 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 07:13:33.824443 20585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:33.824637 20570 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:13:33.824708 20570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "dec504cb31a9433abfc3870c56fd46d3"
format_stamp: "Formatted at 2026-09-24 07:13:33 on dist-test-slave-14ww"
server_key: "6216c8857618f6e494549667860cf510"
server_key_iv: "962e6bd70d282da1b18941c9e72b327e"
server_key_version: "encryptionkey@0"
I20260924 07:13:33.824816 20570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:33.834441 20570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:33.836627 20570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:33.836817 20570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:33.840492 20570 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42551
I20260924 07:13:33.840552 20637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42551 every 8 connection(s)
I20260924 07:13:33.840955 20570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:13:33.841792 20638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:33.843324 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 20570
I20260924 07:13:33.843483 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:13:33.843660 392 external_mini_cluster.cc:1428] Setting key 483ce2af5c32dccebe7ebc4dac26df3a
I20260924 07:13:33.844235 20638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dec504cb31a9433abfc3870c56fd46d3: Bootstrap starting.
I20260924 07:13:33.844856 20638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dec504cb31a9433abfc3870c56fd46d3: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:33.845144 20638 log.cc:824] T 00000000000000000000000000000000 P dec504cb31a9433abfc3870c56fd46d3: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:33.845794 20638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dec504cb31a9433abfc3870c56fd46d3: No bootstrap required, opened a new log
I20260924 07:13:33.847458 20638 raft_consensus.cc:348] T 00000000000000000000000000000000 P dec504cb31a9433abfc3870c56fd46d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec504cb31a9433abfc3870c56fd46d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42551 } }
I20260924 07:13:33.847616 20638 raft_consensus.cc:374] T 00000000000000000000000000000000 P dec504cb31a9433abfc3870c56fd46d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:33.847684 20638 raft_consensus.cc:729] T 00000000000000000000000000000000 P dec504cb31a9433abfc3870c56fd46d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dec504cb31a9433abfc3870c56fd46d3, State: Initialized, Role: FOLLOWER
I20260924 07:13:33.847815 20638 consensus_queue.cc:260] T 00000000000000000000000000000000 P dec504cb31a9433abfc3870c56fd46d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec504cb31a9433abfc3870c56fd46d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42551 } }
I20260924 07:13:33.847905 20638 raft_consensus.cc:388] T 00000000000000000000000000000000 P dec504cb31a9433abfc3870c56fd46d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:13:33.847949 20638 raft_consensus.cc:482] T 00000000000000000000000000000000 P dec504cb31a9433abfc3870c56fd46d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:13:33.847995 20638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dec504cb31a9433abfc3870c56fd46d3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:33.848488 20638 raft_consensus.cc:504] T 00000000000000000000000000000000 P dec504cb31a9433abfc3870c56fd46d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec504cb31a9433abfc3870c56fd46d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42551 } }
I20260924 07:13:33.848601 20638 leader_election.cc:302] T 00000000000000000000000000000000 P dec504cb31a9433abfc3870c56fd46d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dec504cb31a9433abfc3870c56fd46d3; no voters:
I20260924 07:13:33.848803 20638 leader_election.cc:288] T 00000000000000000000000000000000 P dec504cb31a9433abfc3870c56fd46d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:13:33.848824 20643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dec504cb31a9433abfc3870c56fd46d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:13:33.848935 20643 raft_consensus.cc:686] T 00000000000000000000000000000000 P dec504cb31a9433abfc3870c56fd46d3 [term 1 LEADER]: Becoming Leader. State: Replica: dec504cb31a9433abfc3870c56fd46d3, State: Running, Role: LEADER
I20260924 07:13:33.849012 20643 consensus_queue.cc:237] T 00000000000000000000000000000000 P dec504cb31a9433abfc3870c56fd46d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec504cb31a9433abfc3870c56fd46d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42551 } }
I20260924 07:13:33.849047 20638 sys_catalog.cc:564] T 00000000000000000000000000000000 P dec504cb31a9433abfc3870c56fd46d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:13:33.849298 20643 sys_catalog.cc:455] T 00000000000000000000000000000000 P dec504cb31a9433abfc3870c56fd46d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dec504cb31a9433abfc3870c56fd46d3. Latest consensus state: current_term: 1 leader_uuid: "dec504cb31a9433abfc3870c56fd46d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec504cb31a9433abfc3870c56fd46d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42551 } } }
I20260924 07:13:33.849373 20643 sys_catalog.cc:458] T 00000000000000000000000000000000 P dec504cb31a9433abfc3870c56fd46d3 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:33.849311 20644 sys_catalog.cc:455] T 00000000000000000000000000000000 P dec504cb31a9433abfc3870c56fd46d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dec504cb31a9433abfc3870c56fd46d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dec504cb31a9433abfc3870c56fd46d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42551 } } }
I20260924 07:13:33.849563 20644 sys_catalog.cc:458] T 00000000000000000000000000000000 P dec504cb31a9433abfc3870c56fd46d3 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:33.849579 20648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:13:33.850098 20648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 07:13:33 dist-test-slave-14ww krb5kdc[20557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 07:13:33.852095 20648 catalog_manager.cc:1347] Generated new cluster ID: c9811ce9da8844afa16f3629c6865a4a
I20260924 07:13:33.852161 20648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:13:33.854640 20641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:33.844229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35429 (local address 127.0.0.1:42551)
0924 07:13:33.844436 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:33.844451 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:33.844477 (+ 26us) server_negotiation.cc:407] Connection header received
0924 07:13:33.844999 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:33.845008 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:33.845070 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:33.845203 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:33.845849 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:33.846742 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:33.847508 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:33.848083 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:33.848200 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:33.851134 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:33.851138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:33.851138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:33.851150 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:33.853012 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:33.853430 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:33.853433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:33.853433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:33.853536 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:33.853768 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:33.853769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:33.853769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:33.854071 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:33.854231 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:33.854336 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 07:13:33.854433 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":70,"threads_started":1}
I20260924 07:13:33.855669 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:13:33.864248 20641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:33.857130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44311 (local address 127.0.0.1:42551)
0924 07:13:33.857173 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:33.857175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:33.857336 (+ 161us) server_negotiation.cc:407] Connection header received
0924 07:13:33.857405 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:33.857405 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:33.857415 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:33.857464 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:33.858081 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:33.858797 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:33.859795 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:33.860111 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:33.860168 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:33.861536 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:33.861538 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:33.861538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:33.861544 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:33.863052 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:33.863464 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:33.863465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:33.863466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:33.863517 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:33.863737 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:33.863738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:33.863738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:33.863844 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:33.863944 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:33.864104 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 07:13:33.864164 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 07:13:33.874374 20648 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:13:33.874856 20648 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:13:33.879200 20648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dec504cb31a9433abfc3870c56fd46d3: Generated new TSK 0
I20260924 07:13:33.879360 20648 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:13:33.890287 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 20570
W20260924 07:13:33.894630 20555 connection.cc:535] client connection to 127.0.0.1:42551 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42551): Connection reset by peer (error 104)
2026-09-24T07:13:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1314 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:13:33 dist-test-slave-14ww krb5kdc[20669](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:13:33 dist-test-slave-14ww krb5kdc[20669](info): set up 2 sockets
Sep 24 07:13:33 dist-test-slave-14ww krb5kdc[20669](info): commencing operation
krb5kdc: starting...
W20260924 07:13:34.936703 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:13:35 dist-test-slave-14ww krb5kdc[20669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:13:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:13:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:13:35.052126 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42329
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:35.132148 20682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:35.132351 20682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:35.132387 20682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:35.133555 20682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:13:35.133617 20682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:35.133644 20682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:35.133677 20682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:13:35.133701 20682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:13:35.135140 20682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42329
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:35.135380 20682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:35.135599 20682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:35.140280 20688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:35.140398 20687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:35.140281 20690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:35.140448 20682 server_base.cc:1034] running on GCE node
I20260924 07:13:35.140703 20682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:35.140996 20682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:35.142158 20682 hybrid_clock.cc:648] HybridClock initialized: now 1790234015142152 us; error 36 us; skew 500 ppm
Sep 24 07:13:35 dist-test-slave-14ww krb5kdc[20669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234015, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:35.144606 20682 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:13:35.145088 20682 webserver.cc:460] Webserver started at http://127.0.0.1:44173/ using document root <none> and password file <none>
I20260924 07:13:35.145314 20682 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:35.145377 20682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:35.145505 20682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:35.146494 20682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "79c0141db6e14da9b9d3a2d117a9ccdd"
format_stamp: "Formatted at 2026-09-24 07:13:35 on dist-test-slave-14ww"
server_key: "cab7b4622adcce710a41586f08a17ee8"
server_key_iv: "fc82e351e424536da863877d1c2136a7"
server_key_version: "encryptionkey@0"
I20260924 07:13:35.146832 20682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "79c0141db6e14da9b9d3a2d117a9ccdd"
format_stamp: "Formatted at 2026-09-24 07:13:35 on dist-test-slave-14ww"
server_key: "cab7b4622adcce710a41586f08a17ee8"
server_key_iv: "fc82e351e424536da863877d1c2136a7"
server_key_version: "encryptionkey@0"
I20260924 07:13:35.148104 20682 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:13:35.148855 20697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:35.149031 20682 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:13:35.149127 20682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "79c0141db6e14da9b9d3a2d117a9ccdd"
format_stamp: "Formatted at 2026-09-24 07:13:35 on dist-test-slave-14ww"
server_key: "cab7b4622adcce710a41586f08a17ee8"
server_key_iv: "fc82e351e424536da863877d1c2136a7"
server_key_version: "encryptionkey@0"
I20260924 07:13:35.149209 20682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:35.160902 20682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:35.163050 20682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:35.163200 20682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:35.167106 20682 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42329
I20260924 07:13:35.167148 20749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42329 every 8 connection(s)
I20260924 07:13:35.167464 20682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:13:35.168223 20750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:35.170433 20750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79c0141db6e14da9b9d3a2d117a9ccdd: Bootstrap starting.
I20260924 07:13:35.170985 20750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79c0141db6e14da9b9d3a2d117a9ccdd: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:35.171267 20750 log.cc:824] T 00000000000000000000000000000000 P 79c0141db6e14da9b9d3a2d117a9ccdd: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:35.171919 20750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79c0141db6e14da9b9d3a2d117a9ccdd: No bootstrap required, opened a new log
I20260924 07:13:35.173485 20750 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79c0141db6e14da9b9d3a2d117a9ccdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c0141db6e14da9b9d3a2d117a9ccdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42329 } }
I20260924 07:13:35.173609 20750 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79c0141db6e14da9b9d3a2d117a9ccdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:35.173631 20750 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79c0141db6e14da9b9d3a2d117a9ccdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79c0141db6e14da9b9d3a2d117a9ccdd, State: Initialized, Role: FOLLOWER
I20260924 07:13:35.173799 20750 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79c0141db6e14da9b9d3a2d117a9ccdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c0141db6e14da9b9d3a2d117a9ccdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42329 } }
I20260924 07:13:35.173856 20750 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79c0141db6e14da9b9d3a2d117a9ccdd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:13:35.173877 20750 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79c0141db6e14da9b9d3a2d117a9ccdd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:13:35.173946 20750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79c0141db6e14da9b9d3a2d117a9ccdd [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:35.174454 20750 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79c0141db6e14da9b9d3a2d117a9ccdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c0141db6e14da9b9d3a2d117a9ccdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42329 } }
I20260924 07:13:35.174561 20750 leader_election.cc:302] T 00000000000000000000000000000000 P 79c0141db6e14da9b9d3a2d117a9ccdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79c0141db6e14da9b9d3a2d117a9ccdd; no voters:
I20260924 07:13:35.174741 20750 leader_election.cc:288] T 00000000000000000000000000000000 P 79c0141db6e14da9b9d3a2d117a9ccdd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:13:35.174803 20753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79c0141db6e14da9b9d3a2d117a9ccdd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:13:35.174968 20753 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79c0141db6e14da9b9d3a2d117a9ccdd [term 1 LEADER]: Becoming Leader. State: Replica: 79c0141db6e14da9b9d3a2d117a9ccdd, State: Running, Role: LEADER
I20260924 07:13:35.174984 20750 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79c0141db6e14da9b9d3a2d117a9ccdd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:13:35.175107 20753 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79c0141db6e14da9b9d3a2d117a9ccdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c0141db6e14da9b9d3a2d117a9ccdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42329 } }
I20260924 07:13:35.176245 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 20682
I20260924 07:13:35.176234 20755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79c0141db6e14da9b9d3a2d117a9ccdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79c0141db6e14da9b9d3a2d117a9ccdd. Latest consensus state: current_term: 1 leader_uuid: "79c0141db6e14da9b9d3a2d117a9ccdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c0141db6e14da9b9d3a2d117a9ccdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42329 } } }
I20260924 07:13:35.176306 20755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79c0141db6e14da9b9d3a2d117a9ccdd [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:35.176378 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:13:35.176534 20763 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:13:35.176605 392 external_mini_cluster.cc:1428] Setting key e09d9e4800f6e45b206b7245228b54c2
I20260924 07:13:35.176802 20754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79c0141db6e14da9b9d3a2d117a9ccdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79c0141db6e14da9b9d3a2d117a9ccdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79c0141db6e14da9b9d3a2d117a9ccdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42329 } } }
I20260924 07:13:35.176865 20754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79c0141db6e14da9b9d3a2d117a9ccdd [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:35.177130 20763 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:13:35.178469 20763 catalog_manager.cc:1347] Generated new cluster ID: 71ec7f434bbc42b4ade251ce3096cd75
I20260924 07:13:35.178539 20763 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:13:35.186450 20771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:35.177034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32939 (local address 127.0.0.1:42329)
0924 07:13:35.181108 (+ 4074us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:35.181116 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:35.181145 (+ 29us) server_negotiation.cc:407] Connection header received
0924 07:13:35.181246 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:35.181252 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:35.181301 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:35.181416 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:35.181690 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:35.183104 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:35.184247 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:35.184868 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:35.184960 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:35.185724 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:35.185728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:35.185728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:13:35.185936 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:35.185973 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:13:35.185996 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:35.186076 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:35.186193 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 07:13:35.186219 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4025,"thread_start_us":64,"threads_started":1}
I20260924 07:13:35.187211 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:13:35.192088 20771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:35.188658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33661 (local address 127.0.0.1:42329)
0924 07:13:35.188689 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:35.188690 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:35.188811 (+ 121us) server_negotiation.cc:407] Connection header received
0924 07:13:35.188889 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:35.188890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:35.188899 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:35.188925 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:35.189104 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:35.189815 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:35.190785 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:35.191106 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:35.191139 (+ 33us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:35.191829 (+ 690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:35.191830 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:35.191831 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:13:35.191834 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:35.191846 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:13:35.191850 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:35.191878 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:35.192005 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 07:13:35.192018 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260924 07:13:35.211458 20763 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:13:35.212191 20763 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:13:35.219873 20763 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79c0141db6e14da9b9d3a2d117a9ccdd: Generated new TSK 0
I20260924 07:13:35.220062 20763 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:13:35.226392 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 20682
W20260924 07:13:35.231101 20667 connection.cc:535] client connection to 127.0.0.1:42329 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42329): Connection reset by peer (error 104)
2026-09-24T07:13:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:13:35 dist-test-slave-14ww krb5kdc[20781](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:13:35 dist-test-slave-14ww krb5kdc[20781](info): set up 2 sockets
Sep 24 07:13:35 dist-test-slave-14ww krb5kdc[20781](info): commencing operation
krb5kdc: starting...
W20260924 07:13:36.268697 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:13:36 dist-test-slave-14ww krb5kdc[20781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:13:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:13:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:13:36.388072 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40789
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:36.467483 20794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:36.467713 20794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:36.467751 20794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:36.468904 20794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:13:36.468977 20794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:36.469002 20794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:36.469033 20794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:13:36.469055 20794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:13:36.470502 20794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40789
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:36.470762 20794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:36.471037 20794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:36.475878 20799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:36.475886 20802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:36.475914 20794 server_base.cc:1034] running on GCE node
W20260924 07:13:36.476018 20800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:36.476343 20794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:36.476621 20794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:36.477771 20794 hybrid_clock.cc:648] HybridClock initialized: now 1790234016477740 us; error 78 us; skew 500 ppm
Sep 24 07:13:36 dist-test-slave-14ww krb5kdc[20781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234016, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:36.480152 20794 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:13:36.480618 20794 webserver.cc:460] Webserver started at http://127.0.0.1:38009/ using document root <none> and password file <none>
I20260924 07:13:36.480882 20794 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:36.480962 20794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:36.481093 20794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:36.482071 20794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "632cb336824044a882a488ed794202b7"
format_stamp: "Formatted at 2026-09-24 07:13:36 on dist-test-slave-14ww"
server_key: "71cde45ead4d45b88721e2aef948d62c"
server_key_iv: "5f344b96afe91fdb497e40634584b96c"
server_key_version: "encryptionkey@0"
I20260924 07:13:36.482427 20794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "632cb336824044a882a488ed794202b7"
format_stamp: "Formatted at 2026-09-24 07:13:36 on dist-test-slave-14ww"
server_key: "71cde45ead4d45b88721e2aef948d62c"
server_key_iv: "5f344b96afe91fdb497e40634584b96c"
server_key_version: "encryptionkey@0"
I20260924 07:13:36.483852 20794 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:13:36.484596 20809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:36.484798 20794 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:13:36.484908 20794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "632cb336824044a882a488ed794202b7"
format_stamp: "Formatted at 2026-09-24 07:13:36 on dist-test-slave-14ww"
server_key: "71cde45ead4d45b88721e2aef948d62c"
server_key_iv: "5f344b96afe91fdb497e40634584b96c"
server_key_version: "encryptionkey@0"
I20260924 07:13:36.485033 20794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:36.488125 20794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:36.489943 20794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:36.490131 20794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:36.494035 20794 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40789
I20260924 07:13:36.494061 20861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40789 every 8 connection(s)
I20260924 07:13:36.494359 20794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:13:36.495231 20862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:36.497361 20862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 632cb336824044a882a488ed794202b7: Bootstrap starting.
I20260924 07:13:36.497906 20862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 632cb336824044a882a488ed794202b7: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:36.498186 20862 log.cc:824] T 00000000000000000000000000000000 P 632cb336824044a882a488ed794202b7: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:36.498796 20862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 632cb336824044a882a488ed794202b7: No bootstrap required, opened a new log
I20260924 07:13:36.500422 20862 raft_consensus.cc:348] T 00000000000000000000000000000000 P 632cb336824044a882a488ed794202b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "632cb336824044a882a488ed794202b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40789 } }
I20260924 07:13:36.500545 20862 raft_consensus.cc:374] T 00000000000000000000000000000000 P 632cb336824044a882a488ed794202b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:36.500567 20862 raft_consensus.cc:729] T 00000000000000000000000000000000 P 632cb336824044a882a488ed794202b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 632cb336824044a882a488ed794202b7, State: Initialized, Role: FOLLOWER
I20260924 07:13:36.500705 20862 consensus_queue.cc:260] T 00000000000000000000000000000000 P 632cb336824044a882a488ed794202b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "632cb336824044a882a488ed794202b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40789 } }
I20260924 07:13:36.500833 20862 raft_consensus.cc:388] T 00000000000000000000000000000000 P 632cb336824044a882a488ed794202b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:13:36.500877 20862 raft_consensus.cc:482] T 00000000000000000000000000000000 P 632cb336824044a882a488ed794202b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:13:36.500946 20862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 632cb336824044a882a488ed794202b7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:36.501705 20862 raft_consensus.cc:504] T 00000000000000000000000000000000 P 632cb336824044a882a488ed794202b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "632cb336824044a882a488ed794202b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40789 } }
I20260924 07:13:36.501816 20862 leader_election.cc:302] T 00000000000000000000000000000000 P 632cb336824044a882a488ed794202b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 632cb336824044a882a488ed794202b7; no voters:
I20260924 07:13:36.501966 20862 leader_election.cc:288] T 00000000000000000000000000000000 P 632cb336824044a882a488ed794202b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:13:36.502154 20865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 632cb336824044a882a488ed794202b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:13:36.502207 20862 sys_catalog.cc:564] T 00000000000000000000000000000000 P 632cb336824044a882a488ed794202b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:13:36.502521 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 20794
I20260924 07:13:36.502548 20865 raft_consensus.cc:686] T 00000000000000000000000000000000 P 632cb336824044a882a488ed794202b7 [term 1 LEADER]: Becoming Leader. State: Replica: 632cb336824044a882a488ed794202b7, State: Running, Role: LEADER
I20260924 07:13:36.502628 20865 consensus_queue.cc:237] T 00000000000000000000000000000000 P 632cb336824044a882a488ed794202b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "632cb336824044a882a488ed794202b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40789 } }
I20260924 07:13:36.502655 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:13:36.502869 392 external_mini_cluster.cc:1428] Setting key 5be7ce7487676f92ad0bc884d362fc06
I20260924 07:13:36.502964 20866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 632cb336824044a882a488ed794202b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "632cb336824044a882a488ed794202b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "632cb336824044a882a488ed794202b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40789 } } }
I20260924 07:13:36.503046 20866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 632cb336824044a882a488ed794202b7 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:36.503165 20867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 632cb336824044a882a488ed794202b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 632cb336824044a882a488ed794202b7. Latest consensus state: current_term: 1 leader_uuid: "632cb336824044a882a488ed794202b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "632cb336824044a882a488ed794202b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40789 } } }
I20260924 07:13:36.503234 20867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 632cb336824044a882a488ed794202b7 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:36.503423 20874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:13:36.504189 20874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:13:36.505273 20881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:36.503447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37863 (local address 127.0.0.1:40789)
0924 07:13:36.503599 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:36.503607 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:36.503647 (+ 40us) server_negotiation.cc:407] Connection header received
0924 07:13:36.503991 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:36.503998 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:36.504027 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:36.504192 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:36.504861 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:36.504864 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:36.504864 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:13:36.504868 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:36.504903 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:13:36.504913 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:36.504988 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:36.505055 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 07:13:36.505096 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":80,"threads_started":1}
I20260924 07:13:36.505630 20874 catalog_manager.cc:1347] Generated new cluster ID: 564be42b67dd4ba0b66347a560bcd619
I20260924 07:13:36.505681 20874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:13:36.506111 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:13:36.512564 20881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:36.507414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57243 (local address 127.0.0.1:40789)
0924 07:13:36.507450 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:36.507451 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:36.511645 (+ 4194us) server_negotiation.cc:407] Connection header received
0924 07:13:36.511717 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:36.511718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:36.511723 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:36.511767 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:36.512338 (+ 571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:36.512339 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:36.512339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:13:36.512343 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:36.512356 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:13:36.512361 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:36.512410 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:36.512461 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 07:13:36.512475 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 07:13:36.531898 20874 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:13:36.532059 20874 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:13:36.539197 20874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 632cb336824044a882a488ed794202b7: Generated new TSK 0
I20260924 07:13:36.539381 20874 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:13:36.541407 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 20794
2026-09-24T07:13:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1314 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:13:36 dist-test-slave-14ww krb5kdc[20893](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:13:36 dist-test-slave-14ww krb5kdc[20893](info): set up 2 sockets
Sep 24 07:13:36 dist-test-slave-14ww krb5kdc[20893](info): commencing operation
krb5kdc: starting...
W20260924 07:13:37.568940 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:13:37 dist-test-slave-14ww krb5kdc[20893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:13:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:13:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:13:37.683835 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37517
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:37.762420 20906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:37.762601 20906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:37.762624 20906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:37.763667 20906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:13:37.763700 20906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:37.763715 20906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:37.763727 20906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:13:37.763739 20906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:13:37.765090 20906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:37.765265 20906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:37.765417 20906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:37.770393 20912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:37.770535 20911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:37.770385 20914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:37.771430 20906 server_base.cc:1034] running on GCE node
I20260924 07:13:37.771744 20906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:37.772161 20906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:37.773391 20906 hybrid_clock.cc:648] HybridClock initialized: now 1790234017773366 us; error 40 us; skew 500 ppm
Sep 24 07:13:37 dist-test-slave-14ww krb5kdc[20893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234017, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:37.785247 20906 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:13:37.785741 20906 webserver.cc:460] Webserver started at http://127.0.0.1:34999/ using document root <none> and password file <none>
I20260924 07:13:37.785986 20906 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:37.786052 20906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:37.786211 20906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:37.787285 20906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "00e3d8097d0b400ea3458657738a0130"
format_stamp: "Formatted at 2026-09-24 07:13:37 on dist-test-slave-14ww"
server_key: "82472b10daeaf6073bab91c608392fd8"
server_key_iv: "7aa54b7347df4827bb62f1fd9819e046"
server_key_version: "encryptionkey@0"
I20260924 07:13:37.787647 20906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "00e3d8097d0b400ea3458657738a0130"
format_stamp: "Formatted at 2026-09-24 07:13:37 on dist-test-slave-14ww"
server_key: "82472b10daeaf6073bab91c608392fd8"
server_key_iv: "7aa54b7347df4827bb62f1fd9819e046"
server_key_version: "encryptionkey@0"
I20260924 07:13:37.789103 20906 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 07:13:37.789855 20921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:37.790043 20906 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:13:37.790158 20906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "00e3d8097d0b400ea3458657738a0130"
format_stamp: "Formatted at 2026-09-24 07:13:37 on dist-test-slave-14ww"
server_key: "82472b10daeaf6073bab91c608392fd8"
server_key_iv: "7aa54b7347df4827bb62f1fd9819e046"
server_key_version: "encryptionkey@0"
I20260924 07:13:37.790233 20906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:37.809490 20906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:37.811677 20906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:37.811829 20906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:37.815965 20906 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37517
I20260924 07:13:37.816030 20973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37517 every 8 connection(s)
I20260924 07:13:37.816385 20906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:13:37.817294 20974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:37.818553 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 20906
I20260924 07:13:37.818713 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:13:37.818905 392 external_mini_cluster.cc:1428] Setting key a86d013af0c0dc2d1181bbec221305f2
I20260924 07:13:37.819811 20974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00e3d8097d0b400ea3458657738a0130: Bootstrap starting.
I20260924 07:13:37.820420 20974 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00e3d8097d0b400ea3458657738a0130: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:37.820775 20974 log.cc:824] T 00000000000000000000000000000000 P 00e3d8097d0b400ea3458657738a0130: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:37.821506 20974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00e3d8097d0b400ea3458657738a0130: No bootstrap required, opened a new log
I20260924 07:13:37.823252 20974 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00e3d8097d0b400ea3458657738a0130 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e3d8097d0b400ea3458657738a0130" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37517 } }
I20260924 07:13:37.823405 20974 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00e3d8097d0b400ea3458657738a0130 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:37.823457 20974 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00e3d8097d0b400ea3458657738a0130 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00e3d8097d0b400ea3458657738a0130, State: Initialized, Role: FOLLOWER
I20260924 07:13:37.823583 20974 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00e3d8097d0b400ea3458657738a0130 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e3d8097d0b400ea3458657738a0130" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37517 } }
I20260924 07:13:37.823660 20974 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00e3d8097d0b400ea3458657738a0130 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:13:37.823683 20974 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00e3d8097d0b400ea3458657738a0130 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:13:37.823745 20974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00e3d8097d0b400ea3458657738a0130 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:37.824249 20974 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00e3d8097d0b400ea3458657738a0130 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e3d8097d0b400ea3458657738a0130" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37517 } }
I20260924 07:13:37.824369 20974 leader_election.cc:302] T 00000000000000000000000000000000 P 00e3d8097d0b400ea3458657738a0130 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00e3d8097d0b400ea3458657738a0130; no voters:
I20260924 07:13:37.824544 20974 leader_election.cc:288] T 00000000000000000000000000000000 P 00e3d8097d0b400ea3458657738a0130 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:13:37.824568 20979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00e3d8097d0b400ea3458657738a0130 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:13:37.824688 20979 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00e3d8097d0b400ea3458657738a0130 [term 1 LEADER]: Becoming Leader. State: Replica: 00e3d8097d0b400ea3458657738a0130, State: Running, Role: LEADER
I20260924 07:13:37.824823 20974 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00e3d8097d0b400ea3458657738a0130 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:13:37.824841 20979 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00e3d8097d0b400ea3458657738a0130 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e3d8097d0b400ea3458657738a0130" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37517 } }
I20260924 07:13:37.825127 20979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00e3d8097d0b400ea3458657738a0130 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00e3d8097d0b400ea3458657738a0130. Latest consensus state: current_term: 1 leader_uuid: "00e3d8097d0b400ea3458657738a0130" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e3d8097d0b400ea3458657738a0130" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37517 } } }
I20260924 07:13:37.825258 20979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00e3d8097d0b400ea3458657738a0130 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:37.825510 20986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 24 07:13:37 dist-test-slave-14ww krb5kdc[20893](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 07:13:37.826066 20986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:13:37.826844 20980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00e3d8097d0b400ea3458657738a0130 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00e3d8097d0b400ea3458657738a0130" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e3d8097d0b400ea3458657738a0130" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37517 } } }
I20260924 07:13:37.827261 20980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00e3d8097d0b400ea3458657738a0130 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:37.828846 20986 catalog_manager.cc:1347] Generated new cluster ID: d5803504cc12497aa70f07811f69114f
I20260924 07:13:37.828884 20986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:13:37.829998 20977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:37.819489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59381 (local address 127.0.0.1:37517)
0924 07:13:37.819668 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:37.819675 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:37.819698 (+ 23us) server_negotiation.cc:407] Connection header received
0924 07:13:37.820133 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:37.820140 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:37.820187 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:37.820299 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:37.821139 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:37.822055 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:37.822805 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:37.823243 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:37.823354 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:37.826356 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:37.826360 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:37.826360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:37.826373 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:37.828393 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:37.828767 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:37.828771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:37.828771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:37.828875 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:37.829100 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:37.829101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:37.829101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:37.829401 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:37.829551 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:37.829656 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 07:13:37.829790 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
I20260924 07:13:37.830888 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:13:37.834280 20986 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:13:37.834744 20986 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:13:37.839457 20977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:37.832304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56969 (local address 127.0.0.1:37517)
0924 07:13:37.832341 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:37.832343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:37.832454 (+ 111us) server_negotiation.cc:407] Connection header received
0924 07:13:37.832542 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:37.832543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:37.832553 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:37.832581 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:37.833285 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:37.834059 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:37.835094 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:37.835393 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:37.835461 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:37.836768 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:37.836769 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:37.836769 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:37.836775 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:37.838266 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:37.838643 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:37.838643 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:37.838644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:37.838695 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:37.838893 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:37.838894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:37.838895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:37.839007 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:37.839094 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:37.839269 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 07:13:37.839360 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 07:13:37.839461 20986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00e3d8097d0b400ea3458657738a0130: Generated new TSK 0
I20260924 07:13:37.839648 20986 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:13:37.841593 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 20906
2026-09-24T07:13:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1300 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:13:37 dist-test-slave-14ww krb5kdc[21005](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:13:37 dist-test-slave-14ww krb5kdc[21005](info): set up 2 sockets
Sep 24 07:13:37 dist-test-slave-14ww krb5kdc[21005](info): commencing operation
krb5kdc: starting...
W20260924 07:13:38.872718 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:13:38 dist-test-slave-14ww krb5kdc[21005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:13:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:13:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:13:38.996721 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44355
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:39.076406 21018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:39.076597 21018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:39.076620 21018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:39.077767 21018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:13:39.077807 21018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:39.077822 21018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:39.077834 21018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:13:39.077853 21018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:13:39.079221 21018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:39.079391 21018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:39.079550 21018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:39.085304 21024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:39.085330 21026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:39.085330 21018 server_base.cc:1034] running on GCE node
W20260924 07:13:39.085426 21023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:39.085783 21018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:39.086067 21018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:39.087244 21018 hybrid_clock.cc:648] HybridClock initialized: now 1790234019087239 us; error 35 us; skew 500 ppm
Sep 24 07:13:39 dist-test-slave-14ww krb5kdc[21005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234019, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:39.089871 21018 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:13:39.090337 21018 webserver.cc:460] Webserver started at http://127.0.0.1:42955/ using document root <none> and password file <none>
I20260924 07:13:39.090564 21018 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:39.090631 21018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:39.090780 21018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:39.091751 21018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "d0ace3e907de49f5a009f2310254807f"
format_stamp: "Formatted at 2026-09-24 07:13:39 on dist-test-slave-14ww"
server_key: "dfa68dac94f73080cb09db69550a5379"
server_key_iv: "0373e8c09ad9455ed614029502ffac38"
server_key_version: "encryptionkey@0"
I20260924 07:13:39.092092 21018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "d0ace3e907de49f5a009f2310254807f"
format_stamp: "Formatted at 2026-09-24 07:13:39 on dist-test-slave-14ww"
server_key: "dfa68dac94f73080cb09db69550a5379"
server_key_iv: "0373e8c09ad9455ed614029502ffac38"
server_key_version: "encryptionkey@0"
I20260924 07:13:39.093837 21018 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260924 07:13:39.094588 21033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:39.094787 21018 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:13:39.094882 21018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "d0ace3e907de49f5a009f2310254807f"
format_stamp: "Formatted at 2026-09-24 07:13:39 on dist-test-slave-14ww"
server_key: "dfa68dac94f73080cb09db69550a5379"
server_key_iv: "0373e8c09ad9455ed614029502ffac38"
server_key_version: "encryptionkey@0"
I20260924 07:13:39.094965 21018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:39.102398 21018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:39.104681 21018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:39.104875 21018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:39.108570 21018 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44355
I20260924 07:13:39.108618 21085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44355 every 8 connection(s)
I20260924 07:13:39.108958 21018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:13:39.109735 21086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:39.111099 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 21018
I20260924 07:13:39.111222 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:13:39.111388 392 external_mini_cluster.cc:1428] Setting key f58ca786bedd1aaae123f1437f207953
I20260924 07:13:39.112284 21086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0ace3e907de49f5a009f2310254807f: Bootstrap starting.
I20260924 07:13:39.112922 21086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0ace3e907de49f5a009f2310254807f: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:39.113195 21086 log.cc:824] T 00000000000000000000000000000000 P d0ace3e907de49f5a009f2310254807f: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:39.113855 21086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0ace3e907de49f5a009f2310254807f: No bootstrap required, opened a new log
I20260924 07:13:39.116252 21086 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0ace3e907de49f5a009f2310254807f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ace3e907de49f5a009f2310254807f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44355 } }
I20260924 07:13:39.116394 21086 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0ace3e907de49f5a009f2310254807f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:39.116436 21086 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0ace3e907de49f5a009f2310254807f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0ace3e907de49f5a009f2310254807f, State: Initialized, Role: FOLLOWER
I20260924 07:13:39.116593 21086 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0ace3e907de49f5a009f2310254807f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ace3e907de49f5a009f2310254807f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44355 } }
I20260924 07:13:39.116678 21086 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0ace3e907de49f5a009f2310254807f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:13:39.116751 21086 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0ace3e907de49f5a009f2310254807f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:13:39.116806 21086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0ace3e907de49f5a009f2310254807f [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:39.116969 21089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:39.111897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53575 (local address 127.0.0.1:44355)
0924 07:13:39.112054 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:39.112062 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:39.112095 (+ 33us) server_negotiation.cc:407] Connection header received
0924 07:13:39.112521 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:39.112528 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:39.112570 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:39.112695 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:39.112952 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:39.114059 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:39.115136 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:39.115504 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:39.115585 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:39.116221 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:39.116225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:39.116225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:13:39.116370 (+ 145us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:39.116396 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:13:39.116406 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:39.116490 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:39.116558 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 07:13:39.116642 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
I20260924 07:13:39.117517 21086 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0ace3e907de49f5a009f2310254807f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ace3e907de49f5a009f2310254807f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44355 } }
I20260924 07:13:39.117619 21086 leader_election.cc:302] T 00000000000000000000000000000000 P d0ace3e907de49f5a009f2310254807f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0ace3e907de49f5a009f2310254807f; no voters:
I20260924 07:13:39.117776 21086 leader_election.cc:288] T 00000000000000000000000000000000 P d0ace3e907de49f5a009f2310254807f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:13:39.117803 21091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0ace3e907de49f5a009f2310254807f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:13:39.117842 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:13:39.117964 21091 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0ace3e907de49f5a009f2310254807f [term 1 LEADER]: Becoming Leader. State: Replica: d0ace3e907de49f5a009f2310254807f, State: Running, Role: LEADER
I20260924 07:13:39.118057 21086 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0ace3e907de49f5a009f2310254807f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:13:39.118078 21091 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0ace3e907de49f5a009f2310254807f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ace3e907de49f5a009f2310254807f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44355 } }
I20260924 07:13:39.118368 21092 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0ace3e907de49f5a009f2310254807f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0ace3e907de49f5a009f2310254807f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ace3e907de49f5a009f2310254807f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44355 } } }
I20260924 07:13:39.118454 21092 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0ace3e907de49f5a009f2310254807f [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:39.118358 21091 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0ace3e907de49f5a009f2310254807f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0ace3e907de49f5a009f2310254807f. Latest consensus state: current_term: 1 leader_uuid: "d0ace3e907de49f5a009f2310254807f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0ace3e907de49f5a009f2310254807f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44355 } } }
I20260924 07:13:39.118637 21091 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0ace3e907de49f5a009f2310254807f [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:39.118754 21096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:13:39.119249 21096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:13:39.121070 21096 catalog_manager.cc:1347] Generated new cluster ID: 77cef3e9d19f48e5b02b152385294423
I20260924 07:13:39.121119 21096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:13:39.123292 21089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:39.119696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47219 (local address 127.0.0.1:44355)
0924 07:13:39.119727 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:39.119729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:39.120150 (+ 421us) server_negotiation.cc:407] Connection header received
0924 07:13:39.120232 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:39.120233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:39.120243 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:39.120311 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:39.120444 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:39.121157 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:39.122112 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:39.122401 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:39.122474 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:39.123056 (+ 582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:39.123058 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:39.123058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:13:39.123062 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:39.123076 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:13:39.123080 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:39.123139 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:39.123199 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 07:13:39.123220 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 07:13:39.131134 21096 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:13:39.131798 21096 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:13:39.138746 21096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0ace3e907de49f5a009f2310254807f: Generated new TSK 0
I20260924 07:13:39.138906 21096 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:13:39.146137 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 21018
2026-09-24T07:13:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1304 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:13:39 dist-test-slave-14ww krb5kdc[21117](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:13:39 dist-test-slave-14ww krb5kdc[21117](info): set up 2 sockets
Sep 24 07:13:39 dist-test-slave-14ww krb5kdc[21117](info): commencing operation
krb5kdc: starting...
W20260924 07:13:40.188926 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:13:40 dist-test-slave-14ww krb5kdc[21117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:13:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:13:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:13:40.305874 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42103
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:40.384390 21130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:40.384620 21130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:40.384655 21130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:40.385854 21130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:13:40.385926 21130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:40.385952 21130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:40.385982 21130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:13:40.386013 21130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:13:40.387457 21130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:40.387732 21130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:40.387970 21130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:40.393039 21138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:40.393049 21136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:40.393064 21135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:40.393066 21130 server_base.cc:1034] running on GCE node
I20260924 07:13:40.393541 21130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:40.393813 21130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:40.394955 21130 hybrid_clock.cc:648] HybridClock initialized: now 1790234020394936 us; error 36 us; skew 500 ppm
Sep 24 07:13:40 dist-test-slave-14ww krb5kdc[21117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234020, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:40.397396 21130 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:13:40.397910 21130 webserver.cc:460] Webserver started at http://127.0.0.1:42049/ using document root <none> and password file <none>
I20260924 07:13:40.398135 21130 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:40.398202 21130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:40.398379 21130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:40.399415 21130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "a919fb21c2a748e3a739dbd19093cc87"
format_stamp: "Formatted at 2026-09-24 07:13:40 on dist-test-slave-14ww"
server_key: "6cddb6ef193082b1ae0217c41bdfc554"
server_key_iv: "4413edb17c98dc87e64879a0ac28f1c2"
server_key_version: "encryptionkey@0"
I20260924 07:13:40.399775 21130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "a919fb21c2a748e3a739dbd19093cc87"
format_stamp: "Formatted at 2026-09-24 07:13:40 on dist-test-slave-14ww"
server_key: "6cddb6ef193082b1ae0217c41bdfc554"
server_key_iv: "4413edb17c98dc87e64879a0ac28f1c2"
server_key_version: "encryptionkey@0"
I20260924 07:13:40.401197 21130 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:13:40.401917 21145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:40.402129 21130 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:13:40.402230 21130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "a919fb21c2a748e3a739dbd19093cc87"
format_stamp: "Formatted at 2026-09-24 07:13:40 on dist-test-slave-14ww"
server_key: "6cddb6ef193082b1ae0217c41bdfc554"
server_key_iv: "4413edb17c98dc87e64879a0ac28f1c2"
server_key_version: "encryptionkey@0"
I20260924 07:13:40.402314 21130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:40.405877 21130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:40.407832 21130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:40.407987 21130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:40.412060 21130 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42103
I20260924 07:13:40.412091 21197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42103 every 8 connection(s)
I20260924 07:13:40.412390 21130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:13:40.413383 21198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:40.415467 21198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a919fb21c2a748e3a739dbd19093cc87: Bootstrap starting.
I20260924 07:13:40.415966 21198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a919fb21c2a748e3a739dbd19093cc87: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:40.416198 21198 log.cc:824] T 00000000000000000000000000000000 P a919fb21c2a748e3a739dbd19093cc87: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:40.416873 21198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a919fb21c2a748e3a739dbd19093cc87: No bootstrap required, opened a new log
I20260924 07:13:40.418416 21198 raft_consensus.cc:348] T 00000000000000000000000000000000 P a919fb21c2a748e3a739dbd19093cc87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a919fb21c2a748e3a739dbd19093cc87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42103 } }
I20260924 07:13:40.418545 21198 raft_consensus.cc:374] T 00000000000000000000000000000000 P a919fb21c2a748e3a739dbd19093cc87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:40.418567 21198 raft_consensus.cc:729] T 00000000000000000000000000000000 P a919fb21c2a748e3a739dbd19093cc87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a919fb21c2a748e3a739dbd19093cc87, State: Initialized, Role: FOLLOWER
I20260924 07:13:40.418712 21198 consensus_queue.cc:260] T 00000000000000000000000000000000 P a919fb21c2a748e3a739dbd19093cc87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a919fb21c2a748e3a739dbd19093cc87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42103 } }
I20260924 07:13:40.418794 21198 raft_consensus.cc:388] T 00000000000000000000000000000000 P a919fb21c2a748e3a739dbd19093cc87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:13:40.418816 21198 raft_consensus.cc:482] T 00000000000000000000000000000000 P a919fb21c2a748e3a739dbd19093cc87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:13:40.418848 21198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a919fb21c2a748e3a739dbd19093cc87 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:40.419315 21198 raft_consensus.cc:504] T 00000000000000000000000000000000 P a919fb21c2a748e3a739dbd19093cc87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a919fb21c2a748e3a739dbd19093cc87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42103 } }
I20260924 07:13:40.419394 21198 leader_election.cc:302] T 00000000000000000000000000000000 P a919fb21c2a748e3a739dbd19093cc87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a919fb21c2a748e3a739dbd19093cc87; no voters:
I20260924 07:13:40.419509 21198 leader_election.cc:288] T 00000000000000000000000000000000 P a919fb21c2a748e3a739dbd19093cc87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:13:40.419584 21201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a919fb21c2a748e3a739dbd19093cc87 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:13:40.419740 21201 raft_consensus.cc:686] T 00000000000000000000000000000000 P a919fb21c2a748e3a739dbd19093cc87 [term 1 LEADER]: Becoming Leader. State: Replica: a919fb21c2a748e3a739dbd19093cc87, State: Running, Role: LEADER
I20260924 07:13:40.419770 21198 sys_catalog.cc:564] T 00000000000000000000000000000000 P a919fb21c2a748e3a739dbd19093cc87 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:13:40.419828 21201 consensus_queue.cc:237] T 00000000000000000000000000000000 P a919fb21c2a748e3a739dbd19093cc87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a919fb21c2a748e3a739dbd19093cc87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42103 } }
I20260924 07:13:40.419896 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 21130
I20260924 07:13:40.420046 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:13:40.420140 21201 sys_catalog.cc:455] T 00000000000000000000000000000000 P a919fb21c2a748e3a739dbd19093cc87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a919fb21c2a748e3a739dbd19093cc87. Latest consensus state: current_term: 1 leader_uuid: "a919fb21c2a748e3a739dbd19093cc87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a919fb21c2a748e3a739dbd19093cc87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42103 } } }
I20260924 07:13:40.420233 21201 sys_catalog.cc:458] T 00000000000000000000000000000000 P a919fb21c2a748e3a739dbd19093cc87 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:40.420275 392 external_mini_cluster.cc:1428] Setting key 46f79cc5331aa89b84283dee31f5ef7e
I20260924 07:13:40.420149 21202 sys_catalog.cc:455] T 00000000000000000000000000000000 P a919fb21c2a748e3a739dbd19093cc87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a919fb21c2a748e3a739dbd19093cc87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a919fb21c2a748e3a739dbd19093cc87" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42103 } } }
I20260924 07:13:40.420464 21202 sys_catalog.cc:458] T 00000000000000000000000000000000 P a919fb21c2a748e3a739dbd19093cc87 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:40.420568 21205 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:13:40.421176 21205 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:13:40.422763 21205 catalog_manager.cc:1347] Generated new cluster ID: 56e973e546ca43bf9722aa794390268f
I20260924 07:13:40.422816 21205 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:13:40.430542 21207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:40.420717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47095 (local address 127.0.0.1:42103)
0924 07:13:40.421024 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:40.421034 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:40.422032 (+ 998us) server_negotiation.cc:407] Connection header received
0924 07:13:40.422302 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:40.422308 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:40.422347 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:40.422489 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:40.429937 (+ 7448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:40.429941 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:40.429941 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:13:40.429945 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:40.430004 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:13:40.430020 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:40.430053 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:40.430226 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 07:13:40.430289 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":89,"threads_started":1}
I20260924 07:13:40.431524 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:13:40.433930 21207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:40.432949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52249 (local address 127.0.0.1:42103)
0924 07:13:40.432979 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:40.432980 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:40.433119 (+ 139us) server_negotiation.cc:407] Connection header received
0924 07:13:40.433173 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:40.433174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:40.433179 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:40.433235 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:40.433758 (+ 523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:40.433759 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:40.433759 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:13:40.433763 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:40.433774 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:13:40.433778 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:40.433829 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:40.433869 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 07:13:40.433879 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 07:13:40.438587 21205 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:13:40.438735 21205 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:13:40.448778 21205 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a919fb21c2a748e3a739dbd19093cc87: Generated new TSK 0
I20260924 07:13:40.448935 21205 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:13:40.452165 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 21130
2026-09-24T07:13:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1305 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:13:40 dist-test-slave-14ww krb5kdc[21229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:13:40 dist-test-slave-14ww krb5kdc[21229](info): set up 2 sockets
Sep 24 07:13:40 dist-test-slave-14ww krb5kdc[21229](info): commencing operation
krb5kdc: starting...
W20260924 07:13:41.488981 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:13:41 dist-test-slave-14ww krb5kdc[21229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:13:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:13:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:13:41.603856 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34425
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:41.685092 21242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:41.685277 21242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:41.685299 21242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:41.686367 21242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:13:41.686398 21242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:41.686412 21242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:41.686425 21242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:13:41.686437 21242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:13:41.687885 21242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:41.688069 21242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:41.688228 21242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:41.692967 21250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:41.693097 21242 server_base.cc:1034] running on GCE node
W20260924 07:13:41.693002 21248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:41.692975 21247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:41.693444 21242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:41.693674 21242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:41.694860 21242 hybrid_clock.cc:648] HybridClock initialized: now 1790234021694855 us; error 35 us; skew 500 ppm
Sep 24 07:13:41 dist-test-slave-14ww krb5kdc[21229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234021, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:41.697342 21242 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:13:41.697824 21242 webserver.cc:460] Webserver started at http://127.0.0.1:43429/ using document root <none> and password file <none>
I20260924 07:13:41.698051 21242 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:41.698120 21242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:41.698269 21242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:41.699261 21242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "4bb78d1cc9144c76a015471554b87a3d"
format_stamp: "Formatted at 2026-09-24 07:13:41 on dist-test-slave-14ww"
server_key: "52bb9df6e7aa49b83b761aa0057cae2f"
server_key_iv: "67cb024355229e410df0a66f2f735c50"
server_key_version: "encryptionkey@0"
I20260924 07:13:41.699609 21242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "4bb78d1cc9144c76a015471554b87a3d"
format_stamp: "Formatted at 2026-09-24 07:13:41 on dist-test-slave-14ww"
server_key: "52bb9df6e7aa49b83b761aa0057cae2f"
server_key_iv: "67cb024355229e410df0a66f2f735c50"
server_key_version: "encryptionkey@0"
I20260924 07:13:41.700938 21242 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 07:13:41.701663 21257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:41.701820 21242 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:13:41.701916 21242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "4bb78d1cc9144c76a015471554b87a3d"
format_stamp: "Formatted at 2026-09-24 07:13:41 on dist-test-slave-14ww"
server_key: "52bb9df6e7aa49b83b761aa0057cae2f"
server_key_iv: "67cb024355229e410df0a66f2f735c50"
server_key_version: "encryptionkey@0"
I20260924 07:13:41.701992 21242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:41.711206 21242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:41.713357 21242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:41.713507 21242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:41.717445 21242 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34425
I20260924 07:13:41.717478 21309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34425 every 8 connection(s)
I20260924 07:13:41.717796 21242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:13:41.718000 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 21242
I20260924 07:13:41.718129 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:13:41.718314 392 external_mini_cluster.cc:1428] Setting key 7891b7dccd806392115c308a2f568405
I20260924 07:13:41.718792 21310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:41.721902 21310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bb78d1cc9144c76a015471554b87a3d: Bootstrap starting.
I20260924 07:13:41.722553 21310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bb78d1cc9144c76a015471554b87a3d: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:41.722843 21310 log.cc:824] T 00000000000000000000000000000000 P 4bb78d1cc9144c76a015471554b87a3d: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:41.723558 21310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bb78d1cc9144c76a015471554b87a3d: No bootstrap required, opened a new log
Sep 24 07:13:41 dist-test-slave-14ww krb5kdc[21229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 07:13:41.725890 21310 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bb78d1cc9144c76a015471554b87a3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bb78d1cc9144c76a015471554b87a3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34425 } }
I20260924 07:13:41.726016 21310 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4bb78d1cc9144c76a015471554b87a3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:41.726047 21310 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bb78d1cc9144c76a015471554b87a3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bb78d1cc9144c76a015471554b87a3d, State: Initialized, Role: FOLLOWER
I20260924 07:13:41.726267 21310 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bb78d1cc9144c76a015471554b87a3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bb78d1cc9144c76a015471554b87a3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34425 } }
I20260924 07:13:41.726346 21310 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4bb78d1cc9144c76a015471554b87a3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:13:41.726369 21310 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bb78d1cc9144c76a015471554b87a3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:13:41.726415 21310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bb78d1cc9144c76a015471554b87a3d [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:41.726920 21310 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bb78d1cc9144c76a015471554b87a3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bb78d1cc9144c76a015471554b87a3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34425 } }
I20260924 07:13:41.727003 21310 leader_election.cc:302] T 00000000000000000000000000000000 P 4bb78d1cc9144c76a015471554b87a3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bb78d1cc9144c76a015471554b87a3d; no voters:
I20260924 07:13:41.727208 21310 leader_election.cc:288] T 00000000000000000000000000000000 P 4bb78d1cc9144c76a015471554b87a3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:13:41.727257 21315 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bb78d1cc9144c76a015471554b87a3d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:13:41.727391 21315 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bb78d1cc9144c76a015471554b87a3d [term 1 LEADER]: Becoming Leader. State: Replica: 4bb78d1cc9144c76a015471554b87a3d, State: Running, Role: LEADER
I20260924 07:13:41.727463 21310 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bb78d1cc9144c76a015471554b87a3d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:13:41.727533 21315 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bb78d1cc9144c76a015471554b87a3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bb78d1cc9144c76a015471554b87a3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34425 } }
I20260924 07:13:41.727970 21316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bb78d1cc9144c76a015471554b87a3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bb78d1cc9144c76a015471554b87a3d. Latest consensus state: current_term: 1 leader_uuid: "4bb78d1cc9144c76a015471554b87a3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bb78d1cc9144c76a015471554b87a3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34425 } } }
I20260924 07:13:41.728112 21316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bb78d1cc9144c76a015471554b87a3d [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:41.728359 21324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:13:41.728915 21324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:13:41.729894 21313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:41.718835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.68:36795 (local address 127.0.0.1:34425)
0924 07:13:41.718988 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:41.718995 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:41.719019 (+ 24us) server_negotiation.cc:407] Connection header received
0924 07:13:41.719422 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:41.719428 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:41.719471 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:41.719577 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:41.720355 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:41.721312 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:41.722392 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:41.722843 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:41.723001 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:41.725718 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:41.725721 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:41.725721 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:41.725731 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:41.727775 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:41.728269 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:41.728272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:41.728272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:41.728376 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:41.728690 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:41.728691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:41.728691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:41.729070 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:41.729248 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:41.729534 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 07:13:41.729641 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":76,"threads_started":1}
I20260924 07:13:41.729936 21317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bb78d1cc9144c76a015471554b87a3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bb78d1cc9144c76a015471554b87a3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bb78d1cc9144c76a015471554b87a3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34425 } } }
I20260924 07:13:41.730468 21317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bb78d1cc9144c76a015471554b87a3d [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:41.730885 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:13:41.733218 21324 catalog_manager.cc:1347] Generated new cluster ID: 8acbae82f6e848f6bc5d84c834926fc8
I20260924 07:13:41.733271 21324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:13:41.738659 21313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:41.732203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.68:36359 (local address 127.0.0.1:34425)
0924 07:13:41.732252 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:41.732254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:41.732298 (+ 44us) server_negotiation.cc:407] Connection header received
0924 07:13:41.732362 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:41.732363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:41.732372 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:41.732426 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:41.733021 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:41.733638 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:41.734357 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:41.734587 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:41.734659 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:41.736160 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:41.736161 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:41.736162 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:41.736170 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:41.737513 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:41.737873 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:41.737874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:41.737874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:41.737929 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:41.738152 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:41.738153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:41.738154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:41.738258 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:41.738366 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:41.738493 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 07:13:41.738569 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 07:13:41.741849 21324 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:13:41.742527 21324 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:13:41.750572 21324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4bb78d1cc9144c76a015471554b87a3d: Generated new TSK 0
I20260924 07:13:41.750735 21324 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:13:41.757540 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 21242
W20260924 07:13:41.761948 21227 connection.cc:535] client connection to 127.0.0.1:34425 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34425): Connection reset by peer (error 104)
2026-09-24T07:13:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1305 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:13:41 dist-test-slave-14ww krb5kdc[21341](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:13:41 dist-test-slave-14ww krb5kdc[21341](info): set up 2 sockets
Sep 24 07:13:41 dist-test-slave-14ww krb5kdc[21341](info): commencing operation
krb5kdc: starting...
W20260924 07:13:42.800851 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:13:42 dist-test-slave-14ww krb5kdc[21341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:13:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:13:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:13:42.914947 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42537
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:42.994040 21354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:42.994284 21354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:42.994321 21354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:42.995456 21354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:13:42.995543 21354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:42.995568 21354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:42.995590 21354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:13:42.995627 21354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:13:42.997133 21354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:42.997403 21354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:42.997635 21354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:43.002581 21360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:43.002588 21359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:43.002588 21362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:43.003152 21354 server_base.cc:1034] running on GCE node
I20260924 07:13:43.003348 21354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:43.003592 21354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:43.004824 21354 hybrid_clock.cc:648] HybridClock initialized: now 1790234023004808 us; error 24 us; skew 500 ppm
Sep 24 07:13:43 dist-test-slave-14ww krb5kdc[21341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:43.021621 21354 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:13:43.022146 21354 webserver.cc:460] Webserver started at http://127.0.0.1:42105/ using document root <none> and password file <none>
I20260924 07:13:43.022378 21354 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:43.022445 21354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:43.022598 21354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:43.023684 21354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "b1e99cb4b2b24d239244a73b8f00775a"
format_stamp: "Formatted at 2026-09-24 07:13:43 on dist-test-slave-14ww"
server_key: "f17155204f36edab32c3f3f545485caa"
server_key_iv: "1368fc343729c047f3d6cbb81f56f078"
server_key_version: "encryptionkey@0"
I20260924 07:13:43.024052 21354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "b1e99cb4b2b24d239244a73b8f00775a"
format_stamp: "Formatted at 2026-09-24 07:13:43 on dist-test-slave-14ww"
server_key: "f17155204f36edab32c3f3f545485caa"
server_key_iv: "1368fc343729c047f3d6cbb81f56f078"
server_key_version: "encryptionkey@0"
I20260924 07:13:43.025490 21354 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:13:43.026249 21369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:43.026441 21354 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:13:43.026544 21354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "b1e99cb4b2b24d239244a73b8f00775a"
format_stamp: "Formatted at 2026-09-24 07:13:43 on dist-test-slave-14ww"
server_key: "f17155204f36edab32c3f3f545485caa"
server_key_iv: "1368fc343729c047f3d6cbb81f56f078"
server_key_version: "encryptionkey@0"
I20260924 07:13:43.026628 21354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:43.043622 21354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:43.045545 21354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:43.045709 21354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:43.049846 21354 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42537
I20260924 07:13:43.049901 21421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42537 every 8 connection(s)
I20260924 07:13:43.050249 21354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:13:43.051100 21422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:43.053674 21422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1e99cb4b2b24d239244a73b8f00775a: Bootstrap starting.
I20260924 07:13:43.054279 21422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1e99cb4b2b24d239244a73b8f00775a: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:43.054579 21422 log.cc:824] T 00000000000000000000000000000000 P b1e99cb4b2b24d239244a73b8f00775a: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:43.055300 21422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1e99cb4b2b24d239244a73b8f00775a: No bootstrap required, opened a new log
I20260924 07:13:43.057201 21422 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1e99cb4b2b24d239244a73b8f00775a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1e99cb4b2b24d239244a73b8f00775a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42537 } }
I20260924 07:13:43.057341 21422 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1e99cb4b2b24d239244a73b8f00775a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:43.057365 21422 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1e99cb4b2b24d239244a73b8f00775a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1e99cb4b2b24d239244a73b8f00775a, State: Initialized, Role: FOLLOWER
I20260924 07:13:43.057560 21422 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1e99cb4b2b24d239244a73b8f00775a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1e99cb4b2b24d239244a73b8f00775a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42537 } }
I20260924 07:13:43.057646 21422 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1e99cb4b2b24d239244a73b8f00775a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:13:43.057693 21422 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1e99cb4b2b24d239244a73b8f00775a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:13:43.057750 21422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1e99cb4b2b24d239244a73b8f00775a [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:43.058297 21422 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1e99cb4b2b24d239244a73b8f00775a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1e99cb4b2b24d239244a73b8f00775a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42537 } }
I20260924 07:13:43.058414 21422 leader_election.cc:302] T 00000000000000000000000000000000 P b1e99cb4b2b24d239244a73b8f00775a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1e99cb4b2b24d239244a73b8f00775a; no voters:
I20260924 07:13:43.058609 21422 leader_election.cc:288] T 00000000000000000000000000000000 P b1e99cb4b2b24d239244a73b8f00775a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:13:43.058638 21425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1e99cb4b2b24d239244a73b8f00775a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:13:43.058813 21425 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1e99cb4b2b24d239244a73b8f00775a [term 1 LEADER]: Becoming Leader. State: Replica: b1e99cb4b2b24d239244a73b8f00775a, State: Running, Role: LEADER
I20260924 07:13:43.058882 21422 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1e99cb4b2b24d239244a73b8f00775a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:13:43.058945 21425 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1e99cb4b2b24d239244a73b8f00775a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1e99cb4b2b24d239244a73b8f00775a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42537 } }
I20260924 07:13:43.059333 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 21354
I20260924 07:13:43.059319 21426 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1e99cb4b2b24d239244a73b8f00775a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1e99cb4b2b24d239244a73b8f00775a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1e99cb4b2b24d239244a73b8f00775a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42537 } } }
I20260924 07:13:43.059327 21427 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1e99cb4b2b24d239244a73b8f00775a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1e99cb4b2b24d239244a73b8f00775a. Latest consensus state: current_term: 1 leader_uuid: "b1e99cb4b2b24d239244a73b8f00775a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1e99cb4b2b24d239244a73b8f00775a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42537 } } }
I20260924 07:13:43.059410 21426 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1e99cb4b2b24d239244a73b8f00775a [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:43.059422 21427 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1e99cb4b2b24d239244a73b8f00775a [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:43.059484 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:13:43.059597 21429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:13:43.059726 392 external_mini_cluster.cc:1428] Setting key db5b7f0a651cc78118e9d9df6f627680
I20260924 07:13:43.060078 21429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:13:43.061766 21429 catalog_manager.cc:1347] Generated new cluster ID: 8873e6999e6b46a299e7c47bef629d55
I20260924 07:13:43.061821 21429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:13:43.069317 21440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:43.060360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.68:50907 (local address 127.0.0.1:42537)
0924 07:13:43.060722 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:43.060752 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:43.065023 (+ 4271us) server_negotiation.cc:407] Connection header received
0924 07:13:43.065187 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:43.065196 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:43.065271 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:43.065402 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:43.065627 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:43.066567 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:43.067328 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:43.067753 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:43.067869 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:43.068574 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:43.068577 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:43.068578 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:13:43.068800 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:43.068838 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:13:43.068856 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:43.068946 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:43.069050 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 07:13:43.069174 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":88,"threads_started":1}
I20260924 07:13:43.070187 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:13:43.074383 21440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:43.071523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.68:33939 (local address 127.0.0.1:42537)
0924 07:13:43.071558 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:43.071559 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:43.071676 (+ 117us) server_negotiation.cc:407] Connection header received
0924 07:13:43.071747 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:43.071748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:43.071758 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:43.071814 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:43.071944 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:43.072384 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:43.073169 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:43.073393 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:43.073474 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:43.074145 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:43.074146 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:43.074146 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:13:43.074150 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:43.074162 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:13:43.074166 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:43.074213 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:43.074283 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 07:13:43.074294 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 07:13:43.090281 21429 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:13:43.090991 21429 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:13:43.099434 21429 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1e99cb4b2b24d239244a73b8f00775a: Generated new TSK 0
I20260924 07:13:43.099623 21429 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:13:43.112142 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 21354
W20260924 07:13:43.116261 21339 connection.cc:535] client connection to 127.0.0.1:42537 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42537): Connection reset by peer (error 104)
2026-09-24T07:13:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1354 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:13:43 dist-test-slave-14ww krb5kdc[21453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:13:43 dist-test-slave-14ww krb5kdc[21453](info): set up 2 sockets
Sep 24 07:13:43 dist-test-slave-14ww krb5kdc[21453](info): commencing operation
krb5kdc: starting...
W20260924 07:13:44.158205 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:13:44 dist-test-slave-14ww krb5kdc[21453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:13:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:13:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:13:44.277859 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34497
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:44.356853 21466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:44.357040 21466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:44.357062 21466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:44.358145 21466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:13:44.358182 21466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:44.358196 21466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:44.358209 21466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:13:44.358222 21466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:13:44.359562 21466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:44.359745 21466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:44.359899 21466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:44.364537 21472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:44.364545 21471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:44.364545 21474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:44.364776 21466 server_base.cc:1034] running on GCE node
I20260924 07:13:44.365020 21466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:44.365291 21466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:44.366456 21466 hybrid_clock.cc:648] HybridClock initialized: now 1790234024366447 us; error 42 us; skew 500 ppm
Sep 24 07:13:44 dist-test-slave-14ww krb5kdc[21453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:44.368917 21466 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:13:44.369356 21466 webserver.cc:460] Webserver started at http://127.0.0.1:40903/ using document root <none> and password file <none>
I20260924 07:13:44.369572 21466 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:44.369637 21466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:44.369773 21466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:44.370754 21466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "c16bf8b06e534383b8057b7047db2d41"
format_stamp: "Formatted at 2026-09-24 07:13:44 on dist-test-slave-14ww"
server_key: "f9ea169c4ecf1138e34bbaa1b4c2d208"
server_key_iv: "fffc3e83310987fd658be091207d21f6"
server_key_version: "encryptionkey@0"
I20260924 07:13:44.371104 21466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "c16bf8b06e534383b8057b7047db2d41"
format_stamp: "Formatted at 2026-09-24 07:13:44 on dist-test-slave-14ww"
server_key: "f9ea169c4ecf1138e34bbaa1b4c2d208"
server_key_iv: "fffc3e83310987fd658be091207d21f6"
server_key_version: "encryptionkey@0"
I20260924 07:13:44.372401 21466 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:13:44.373296 21481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:44.373538 21466 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 07:13:44.373631 21466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "c16bf8b06e534383b8057b7047db2d41"
format_stamp: "Formatted at 2026-09-24 07:13:44 on dist-test-slave-14ww"
server_key: "f9ea169c4ecf1138e34bbaa1b4c2d208"
server_key_iv: "fffc3e83310987fd658be091207d21f6"
server_key_version: "encryptionkey@0"
I20260924 07:13:44.373721 21466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:44.376852 21466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:44.379048 21466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:44.379201 21466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:44.383164 21466 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34497
I20260924 07:13:44.383190 21533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34497 every 8 connection(s)
I20260924 07:13:44.383491 21466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:13:44.384433 21534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:44.386544 21534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c16bf8b06e534383b8057b7047db2d41: Bootstrap starting.
I20260924 07:13:44.387128 21534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c16bf8b06e534383b8057b7047db2d41: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:44.387408 21534 log.cc:824] T 00000000000000000000000000000000 P c16bf8b06e534383b8057b7047db2d41: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:44.388059 21534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c16bf8b06e534383b8057b7047db2d41: No bootstrap required, opened a new log
I20260924 07:13:44.389669 21534 raft_consensus.cc:348] T 00000000000000000000000000000000 P c16bf8b06e534383b8057b7047db2d41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c16bf8b06e534383b8057b7047db2d41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34497 } }
I20260924 07:13:44.389791 21534 raft_consensus.cc:374] T 00000000000000000000000000000000 P c16bf8b06e534383b8057b7047db2d41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:44.389813 21534 raft_consensus.cc:729] T 00000000000000000000000000000000 P c16bf8b06e534383b8057b7047db2d41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c16bf8b06e534383b8057b7047db2d41, State: Initialized, Role: FOLLOWER
I20260924 07:13:44.389974 21534 consensus_queue.cc:260] T 00000000000000000000000000000000 P c16bf8b06e534383b8057b7047db2d41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c16bf8b06e534383b8057b7047db2d41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34497 } }
I20260924 07:13:44.390033 21534 raft_consensus.cc:388] T 00000000000000000000000000000000 P c16bf8b06e534383b8057b7047db2d41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:13:44.390070 21534 raft_consensus.cc:482] T 00000000000000000000000000000000 P c16bf8b06e534383b8057b7047db2d41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:13:44.390127 21534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c16bf8b06e534383b8057b7047db2d41 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:44.390619 21534 raft_consensus.cc:504] T 00000000000000000000000000000000 P c16bf8b06e534383b8057b7047db2d41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c16bf8b06e534383b8057b7047db2d41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34497 } }
I20260924 07:13:44.390723 21534 leader_election.cc:302] T 00000000000000000000000000000000 P c16bf8b06e534383b8057b7047db2d41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c16bf8b06e534383b8057b7047db2d41; no voters:
I20260924 07:13:44.390897 21534 leader_election.cc:288] T 00000000000000000000000000000000 P c16bf8b06e534383b8057b7047db2d41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:13:44.390952 21537 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c16bf8b06e534383b8057b7047db2d41 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:13:44.391140 21534 sys_catalog.cc:564] T 00000000000000000000000000000000 P c16bf8b06e534383b8057b7047db2d41 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:13:44.391140 21537 raft_consensus.cc:686] T 00000000000000000000000000000000 P c16bf8b06e534383b8057b7047db2d41 [term 1 LEADER]: Becoming Leader. State: Replica: c16bf8b06e534383b8057b7047db2d41, State: Running, Role: LEADER
I20260924 07:13:44.391324 21537 consensus_queue.cc:237] T 00000000000000000000000000000000 P c16bf8b06e534383b8057b7047db2d41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c16bf8b06e534383b8057b7047db2d41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34497 } }
I20260924 07:13:44.391671 21538 sys_catalog.cc:455] T 00000000000000000000000000000000 P c16bf8b06e534383b8057b7047db2d41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c16bf8b06e534383b8057b7047db2d41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c16bf8b06e534383b8057b7047db2d41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34497 } } }
I20260924 07:13:44.391772 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 21466
I20260924 07:13:44.391804 21538 sys_catalog.cc:458] T 00000000000000000000000000000000 P c16bf8b06e534383b8057b7047db2d41 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:44.391891 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:13:44.392089 392 external_mini_cluster.cc:1428] Setting key d3c03cb664e53b12c961908b9ee8f822
I20260924 07:13:44.392308 21541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:13:44.392490 21539 sys_catalog.cc:455] T 00000000000000000000000000000000 P c16bf8b06e534383b8057b7047db2d41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c16bf8b06e534383b8057b7047db2d41. Latest consensus state: current_term: 1 leader_uuid: "c16bf8b06e534383b8057b7047db2d41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c16bf8b06e534383b8057b7047db2d41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34497 } } }
I20260924 07:13:44.392632 21539 sys_catalog.cc:458] T 00000000000000000000000000000000 P c16bf8b06e534383b8057b7047db2d41 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:44.392882 21541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:13:44.394510 21541 catalog_manager.cc:1347] Generated new cluster ID: dbdda405c688444c8751c5f0efe9222e
I20260924 07:13:44.394608 21541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:13:44.394797 21555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:44.392706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.68:51935 (local address 127.0.0.1:34497)
0924 07:13:44.392894 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:44.392903 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:44.393347 (+ 444us) server_negotiation.cc:407] Connection header received
0924 07:13:44.393505 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:44.393512 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:44.393542 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:44.393696 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:44.394350 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:44.394354 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:44.394354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:13:44.394359 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:44.394400 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:13:44.394415 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:44.394487 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:44.394558 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 07:13:44.394599 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":90,"threads_started":1}
I20260924 07:13:44.395735 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:13:44.409912 21555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:44.397294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.68:57223 (local address 127.0.0.1:34497)
0924 07:13:44.397326 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:44.397329 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:44.408939 (+ 11610us) server_negotiation.cc:407] Connection header received
0924 07:13:44.409015 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:44.409016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:44.409021 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:44.409072 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:44.409698 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:44.409700 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:44.409700 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:13:44.409704 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:44.409720 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:13:44.409725 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:44.409785 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:44.409833 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 07:13:44.409847 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 07:13:44.424916 21541 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:13:44.425073 21541 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:13:44.434329 21541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c16bf8b06e534383b8057b7047db2d41: Generated new TSK 0
I20260924 07:13:44.434491 21541 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:13:44.436254 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 21466
2026-09-24T07:13:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:13:44 dist-test-slave-14ww krb5kdc[21565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:13:44 dist-test-slave-14ww krb5kdc[21565](info): set up 2 sockets
Sep 24 07:13:44 dist-test-slave-14ww krb5kdc[21565](info): commencing operation
krb5kdc: starting...
W20260924 07:13:45.468847 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:13:45 dist-test-slave-14ww krb5kdc[21565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:13:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:13:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:13:45.587145 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46651
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:45.666478 21578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:45.666682 21578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:45.666703 21578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:45.667750 21578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:13:45.667783 21578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:45.667797 21578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:45.667809 21578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:13:45.667821 21578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:13:45.669251 21578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46651
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:45.669427 21578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:45.669579 21578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:45.674472 21584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:45.674546 21583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:45.674536 21586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:45.674679 21578 server_base.cc:1034] running on GCE node
I20260924 07:13:45.674937 21578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:45.675203 21578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:45.676524 21578 hybrid_clock.cc:648] HybridClock initialized: now 1790234025676516 us; error 39 us; skew 500 ppm
Sep 24 07:13:45 dist-test-slave-14ww krb5kdc[21565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:45.679018 21578 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:13:45.679455 21578 webserver.cc:460] Webserver started at http://127.0.0.1:43435/ using document root <none> and password file <none>
I20260924 07:13:45.679674 21578 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:45.679718 21578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:45.679844 21578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:45.680943 21578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "6bf496d9170c4b39885debe4ac78a950"
format_stamp: "Formatted at 2026-09-24 07:13:45 on dist-test-slave-14ww"
server_key: "db49f77f59c05aa48191deaf8aa83e6d"
server_key_iv: "6f83533419bb89578e67beb19b9792e8"
server_key_version: "encryptionkey@0"
I20260924 07:13:45.681308 21578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "6bf496d9170c4b39885debe4ac78a950"
format_stamp: "Formatted at 2026-09-24 07:13:45 on dist-test-slave-14ww"
server_key: "db49f77f59c05aa48191deaf8aa83e6d"
server_key_iv: "6f83533419bb89578e67beb19b9792e8"
server_key_version: "encryptionkey@0"
I20260924 07:13:45.682595 21578 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:13:45.683297 21593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:45.683486 21578 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:13:45.683586 21578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "6bf496d9170c4b39885debe4ac78a950"
format_stamp: "Formatted at 2026-09-24 07:13:45 on dist-test-slave-14ww"
server_key: "db49f77f59c05aa48191deaf8aa83e6d"
server_key_iv: "6f83533419bb89578e67beb19b9792e8"
server_key_version: "encryptionkey@0"
I20260924 07:13:45.683670 21578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:45.695530 21578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:45.697342 21578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:45.697496 21578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:45.701290 21578 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46651
I20260924 07:13:45.701349 21645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46651 every 8 connection(s)
I20260924 07:13:45.701740 21578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:13:45.702601 21646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:45.705020 21646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bf496d9170c4b39885debe4ac78a950: Bootstrap starting.
I20260924 07:13:45.705618 21646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bf496d9170c4b39885debe4ac78a950: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:45.705892 21646 log.cc:824] T 00000000000000000000000000000000 P 6bf496d9170c4b39885debe4ac78a950: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:45.706553 21646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bf496d9170c4b39885debe4ac78a950: No bootstrap required, opened a new log
I20260924 07:13:45.708107 21646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bf496d9170c4b39885debe4ac78a950 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf496d9170c4b39885debe4ac78a950" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46651 } }
I20260924 07:13:45.708236 21646 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bf496d9170c4b39885debe4ac78a950 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:45.708258 21646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bf496d9170c4b39885debe4ac78a950 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bf496d9170c4b39885debe4ac78a950, State: Initialized, Role: FOLLOWER
I20260924 07:13:45.708401 21646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bf496d9170c4b39885debe4ac78a950 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf496d9170c4b39885debe4ac78a950" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46651 } }
I20260924 07:13:45.708463 21646 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bf496d9170c4b39885debe4ac78a950 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:13:45.708483 21646 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bf496d9170c4b39885debe4ac78a950 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:13:45.708509 21646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bf496d9170c4b39885debe4ac78a950 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:45.709106 21646 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bf496d9170c4b39885debe4ac78a950 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf496d9170c4b39885debe4ac78a950" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46651 } }
I20260924 07:13:45.709215 21646 leader_election.cc:302] T 00000000000000000000000000000000 P 6bf496d9170c4b39885debe4ac78a950 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bf496d9170c4b39885debe4ac78a950; no voters:
I20260924 07:13:45.709393 21646 leader_election.cc:288] T 00000000000000000000000000000000 P 6bf496d9170c4b39885debe4ac78a950 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:13:45.709447 21649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bf496d9170c4b39885debe4ac78a950 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:13:45.709622 21649 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bf496d9170c4b39885debe4ac78a950 [term 1 LEADER]: Becoming Leader. State: Replica: 6bf496d9170c4b39885debe4ac78a950, State: Running, Role: LEADER
I20260924 07:13:45.709641 21646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bf496d9170c4b39885debe4ac78a950 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:13:45.709780 21649 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bf496d9170c4b39885debe4ac78a950 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf496d9170c4b39885debe4ac78a950" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46651 } }
I20260924 07:13:45.710152 21651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bf496d9170c4b39885debe4ac78a950 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bf496d9170c4b39885debe4ac78a950. Latest consensus state: current_term: 1 leader_uuid: "6bf496d9170c4b39885debe4ac78a950" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf496d9170c4b39885debe4ac78a950" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46651 } } }
I20260924 07:13:45.710264 21651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bf496d9170c4b39885debe4ac78a950 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:45.710150 21650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bf496d9170c4b39885debe4ac78a950 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bf496d9170c4b39885debe4ac78a950" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bf496d9170c4b39885debe4ac78a950" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46651 } } }
I20260924 07:13:45.710310 21650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bf496d9170c4b39885debe4ac78a950 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:45.710475 21653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:13:45.711047 21653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:13:45.711899 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 21578
I20260924 07:13:45.712028 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:13:45.712247 392 external_mini_cluster.cc:1428] Setting key f163dd5573ea708eabbbf485a0821447
I20260924 07:13:45.712560 21653 catalog_manager.cc:1347] Generated new cluster ID: f4976fbb740746d8976301670b7d579a
I20260924 07:13:45.712611 21653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:13:45.718801 21653 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:13:45.719590 21653 catalog_manager.cc:1504] Loading token signing keys...
Sep 24 07:13:45 dist-test-slave-14ww krb5kdc[21565](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 07:13:45.727396 21669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:45.712897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.68:54891 (local address 127.0.0.1:46651)
0924 07:13:45.716867 (+ 3970us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:45.716876 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:45.716912 (+ 36us) server_negotiation.cc:407] Connection header received
0924 07:13:45.717021 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:45.717029 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:45.717086 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:45.717239 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:45.717973 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:45.719459 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:45.720480 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:45.721115 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:45.721223 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:45.723950 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:45.723953 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:45.723954 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:45.723965 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:45.725828 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:45.726272 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:45.726274 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:45.726275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:45.726368 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:45.726611 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:45.726612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:45.726612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:45.726853 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:45.726980 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:45.727114 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 07:13:45.727201 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3917,"thread_start_us":81,"threads_started":1}
I20260924 07:13:45.728425 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:13:45.737071 21669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:45.729802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.68:43763 (local address 127.0.0.1:46651)
0924 07:13:45.729831 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:45.729833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:45.729958 (+ 125us) server_negotiation.cc:407] Connection header received
0924 07:13:45.730010 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:45.730011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:45.730019 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:45.730073 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:45.730659 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:45.731573 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:45.732594 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:45.732940 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:45.733001 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:45.734309 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:45.734310 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:45.734310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:45.734317 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:45.735828 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:45.736282 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:45.736283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:45.736283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:45.736334 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:45.736566 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:45.736567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:45.736567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:45.736673 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:45.736800 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:45.736936 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 07:13:45.736991 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 07:13:45.739073 21653 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bf496d9170c4b39885debe4ac78a950: Generated new TSK 0
I20260924 07:13:45.739229 21653 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:13:45.741422 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 21578
W20260924 07:13:45.745303 21563 connection.cc:535] client connection to 127.0.0.1:46651 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46651): Connection reset by peer (error 104)
2026-09-24T07:13:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1304 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:13:45 dist-test-slave-14ww krb5kdc[21677](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:13:45 dist-test-slave-14ww krb5kdc[21677](info): set up 2 sockets
Sep 24 07:13:45 dist-test-slave-14ww krb5kdc[21677](info): commencing operation
krb5kdc: starting...
W20260924 07:13:46.785182 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:13:46 dist-test-slave-14ww krb5kdc[21677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:13:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:13:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:13:46.903918 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45573
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:46.984937 21690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:46.985188 21690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:46.985227 21690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:46.986367 21690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:13:46.986436 21690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:46.986460 21690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:46.986487 21690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:13:46.986519 21690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:13:46.987927 21690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45573
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:46.988198 21690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:46.988420 21690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:46.993522 21696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:46.993532 21695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:46.993661 21690 server_base.cc:1034] running on GCE node
W20260924 07:13:46.993592 21698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:46.994055 21690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:46.994359 21690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:46.995515 21690 hybrid_clock.cc:648] HybridClock initialized: now 1790234026995500 us; error 29 us; skew 500 ppm
Sep 24 07:13:46 dist-test-slave-14ww krb5kdc[21677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:46.998281 21690 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:13:46.998726 21690 webserver.cc:460] Webserver started at http://127.0.0.1:39645/ using document root <none> and password file <none>
I20260924 07:13:46.999037 21690 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:46.999114 21690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:46.999265 21690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:47.000345 21690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "7da372ff188f48108724355e5b695f29"
format_stamp: "Formatted at 2026-09-24 07:13:46 on dist-test-slave-14ww"
server_key: "6cc9f8d299533693551e4e31954bb38b"
server_key_iv: "c7b71c8f0c4e97d381cb4acf366dec2b"
server_key_version: "encryptionkey@0"
I20260924 07:13:47.000712 21690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "7da372ff188f48108724355e5b695f29"
format_stamp: "Formatted at 2026-09-24 07:13:46 on dist-test-slave-14ww"
server_key: "6cc9f8d299533693551e4e31954bb38b"
server_key_iv: "c7b71c8f0c4e97d381cb4acf366dec2b"
server_key_version: "encryptionkey@0"
I20260924 07:13:47.002092 21690 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:13:47.002866 21705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:47.003036 21690 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:13:47.003144 21690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "7da372ff188f48108724355e5b695f29"
format_stamp: "Formatted at 2026-09-24 07:13:46 on dist-test-slave-14ww"
server_key: "6cc9f8d299533693551e4e31954bb38b"
server_key_iv: "c7b71c8f0c4e97d381cb4acf366dec2b"
server_key_version: "encryptionkey@0"
I20260924 07:13:47.003221 21690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:47.015141 21690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:47.017285 21690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:47.017448 21690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:47.021387 21690 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45573
I20260924 07:13:47.021420 21757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45573 every 8 connection(s)
I20260924 07:13:47.021790 21690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:13:47.022787 21758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:47.025058 21758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7da372ff188f48108724355e5b695f29: Bootstrap starting.
I20260924 07:13:47.025573 21758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7da372ff188f48108724355e5b695f29: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:47.025807 21758 log.cc:824] T 00000000000000000000000000000000 P 7da372ff188f48108724355e5b695f29: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:47.026449 21758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7da372ff188f48108724355e5b695f29: No bootstrap required, opened a new log
I20260924 07:13:47.027926 21758 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7da372ff188f48108724355e5b695f29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7da372ff188f48108724355e5b695f29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45573 } }
I20260924 07:13:47.028057 21758 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7da372ff188f48108724355e5b695f29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:47.028079 21758 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7da372ff188f48108724355e5b695f29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da372ff188f48108724355e5b695f29, State: Initialized, Role: FOLLOWER
I20260924 07:13:47.028221 21758 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7da372ff188f48108724355e5b695f29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7da372ff188f48108724355e5b695f29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45573 } }
I20260924 07:13:47.028296 21758 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7da372ff188f48108724355e5b695f29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:13:47.028318 21758 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7da372ff188f48108724355e5b695f29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:13:47.028345 21758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7da372ff188f48108724355e5b695f29 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:47.028574 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 21690
I20260924 07:13:47.028717 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:13:47.028921 392 external_mini_cluster.cc:1428] Setting key 46e3d2f8b3791cb97f34641bbf6199a1
I20260924 07:13:47.028946 21758 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7da372ff188f48108724355e5b695f29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7da372ff188f48108724355e5b695f29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45573 } }
I20260924 07:13:47.029064 21758 leader_election.cc:302] T 00000000000000000000000000000000 P 7da372ff188f48108724355e5b695f29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7da372ff188f48108724355e5b695f29; no voters:
I20260924 07:13:47.029250 21758 leader_election.cc:288] T 00000000000000000000000000000000 P 7da372ff188f48108724355e5b695f29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:13:47.029278 21761 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7da372ff188f48108724355e5b695f29 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:13:47.029434 21761 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7da372ff188f48108724355e5b695f29 [term 1 LEADER]: Becoming Leader. State: Replica: 7da372ff188f48108724355e5b695f29, State: Running, Role: LEADER
I20260924 07:13:47.029508 21758 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7da372ff188f48108724355e5b695f29 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:13:47.029569 21761 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7da372ff188f48108724355e5b695f29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7da372ff188f48108724355e5b695f29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45573 } }
I20260924 07:13:47.029954 21761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7da372ff188f48108724355e5b695f29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7da372ff188f48108724355e5b695f29. Latest consensus state: current_term: 1 leader_uuid: "7da372ff188f48108724355e5b695f29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7da372ff188f48108724355e5b695f29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45573 } } }
I20260924 07:13:47.030105 21762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7da372ff188f48108724355e5b695f29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7da372ff188f48108724355e5b695f29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7da372ff188f48108724355e5b695f29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45573 } } }
I20260924 07:13:47.030194 21762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7da372ff188f48108724355e5b695f29 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:47.030102 21761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7da372ff188f48108724355e5b695f29 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:47.030509 21772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:13:47.031075 21772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:13:47.033407 21772 catalog_manager.cc:1347] Generated new cluster ID: 02a2be368e1d436a965cf8fb936da2f3
I20260924 07:13:47.033459 21772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:13:47.035415 21765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:47.029522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.68:54527 (local address 127.0.0.1:45573)
0924 07:13:47.030183 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:47.030192 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:47.030235 (+ 43us) server_negotiation.cc:407] Connection header received
0924 07:13:47.030516 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:47.030523 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:47.030588 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:47.031027 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:47.031118 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:47.032158 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:47.033258 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:47.033865 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:47.033964 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:47.034739 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:47.034742 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:47.034743 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:13:47.034954 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:47.034990 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:13:47.035003 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:47.035079 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:47.035197 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 07:13:47.035227 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":62,"threads_started":1}
I20260924 07:13:47.036193 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:13:47.042192 21765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:47.038729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.68:54989 (local address 127.0.0.1:45573)
0924 07:13:47.038761 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:47.038762 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:47.038902 (+ 140us) server_negotiation.cc:407] Connection header received
0924 07:13:47.038960 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:47.038961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:47.038972 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:47.039031 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:47.039156 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:47.039849 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:47.040897 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:47.041207 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:47.041268 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:47.041940 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:47.041941 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:47.041941 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:13:47.041945 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:47.041959 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:13:47.041963 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:47.042016 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:47.042108 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 07:13:47.042121 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 07:13:47.060333 21772 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:13:47.061077 21772 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:13:47.067708 21772 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7da372ff188f48108724355e5b695f29: Generated new TSK 0
I20260924 07:13:47.067905 21772 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:13:47.075790 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 21690
W20260924 07:13:47.080173 21675 connection.cc:535] client connection to 127.0.0.1:45573 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45573): Connection reset by peer (error 104)
2026-09-24T07:13:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1334 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:13:47 dist-test-slave-14ww krb5kdc[21789](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:13:47 dist-test-slave-14ww krb5kdc[21789](info): set up 2 sockets
Sep 24 07:13:47 dist-test-slave-14ww krb5kdc[21789](info): commencing operation
krb5kdc: starting...
W20260924 07:13:48.108916 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:13:48 dist-test-slave-14ww krb5kdc[21789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:13:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:13:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 07:13:48.225375 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43105
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:48.303831 21802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:48.304071 21802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:48.304108 21802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:48.305294 21802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:13:48.305361 21802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:48.305384 21802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:48.305421 21802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:13:48.305454 21802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:13:48.306897 21802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43105
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233772407728-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:48.307168 21802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:48.307401 21802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:48.312413 21807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:48.312479 21808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:48.312424 21810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:48.312503 21802 server_base.cc:1034] running on GCE node
I20260924 07:13:48.312870 21802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:48.313159 21802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:48.314298 21802 hybrid_clock.cc:648] HybridClock initialized: now 1790234028314281 us; error 41 us; skew 500 ppm
Sep 24 07:13:48 dist-test-slave-14ww krb5kdc[21789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:48.328609 21802 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 07:13:48.329142 21802 webserver.cc:460] Webserver started at http://127.0.0.1:39769/ using document root <none> and password file <none>
I20260924 07:13:48.329375 21802 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:48.329421 21802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:48.329569 21802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:48.330662 21802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "cd84d1d369b64a948294acb15c960ba7"
format_stamp: "Formatted at 2026-09-24 07:13:48 on dist-test-slave-14ww"
server_key: "377a5d60804b4cabf5c8a10999c7e97f"
server_key_iv: "894a2ffd239d90887e597e4e1e65888e"
server_key_version: "encryptionkey@0"
I20260924 07:13:48.331013 21802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "cd84d1d369b64a948294acb15c960ba7"
format_stamp: "Formatted at 2026-09-24 07:13:48 on dist-test-slave-14ww"
server_key: "377a5d60804b4cabf5c8a10999c7e97f"
server_key_iv: "894a2ffd239d90887e597e4e1e65888e"
server_key_version: "encryptionkey@0"
I20260924 07:13:48.332299 21802 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:13:48.333058 21817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:48.333248 21802 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:13:48.333354 21802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "cd84d1d369b64a948294acb15c960ba7"
format_stamp: "Formatted at 2026-09-24 07:13:48 on dist-test-slave-14ww"
server_key: "377a5d60804b4cabf5c8a10999c7e97f"
server_key_iv: "894a2ffd239d90887e597e4e1e65888e"
server_key_version: "encryptionkey@0"
I20260924 07:13:48.333426 21802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:48.336622 21802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:48.338419 21802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:48.338562 21802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:48.342423 21802 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43105
I20260924 07:13:48.342449 21869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43105 every 8 connection(s)
I20260924 07:13:48.342772 21802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:13:48.343662 21870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:48.345817 21870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd84d1d369b64a948294acb15c960ba7: Bootstrap starting.
I20260924 07:13:48.346309 21870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd84d1d369b64a948294acb15c960ba7: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:48.346526 21870 log.cc:824] T 00000000000000000000000000000000 P cd84d1d369b64a948294acb15c960ba7: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:48.347117 21870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd84d1d369b64a948294acb15c960ba7: No bootstrap required, opened a new log
I20260924 07:13:48.348590 21870 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd84d1d369b64a948294acb15c960ba7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd84d1d369b64a948294acb15c960ba7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } }
I20260924 07:13:48.348711 21870 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd84d1d369b64a948294acb15c960ba7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:48.348789 21870 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd84d1d369b64a948294acb15c960ba7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd84d1d369b64a948294acb15c960ba7, State: Initialized, Role: FOLLOWER
I20260924 07:13:48.348927 21870 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd84d1d369b64a948294acb15c960ba7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd84d1d369b64a948294acb15c960ba7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } }
I20260924 07:13:48.349006 21870 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd84d1d369b64a948294acb15c960ba7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:13:48.349028 21870 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd84d1d369b64a948294acb15c960ba7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:13:48.349056 21870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd84d1d369b64a948294acb15c960ba7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:48.349423 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 21802
I20260924 07:13:48.349542 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:13:48.349670 21870 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd84d1d369b64a948294acb15c960ba7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd84d1d369b64a948294acb15c960ba7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } }
I20260924 07:13:48.349740 392 external_mini_cluster.cc:1428] Setting key 1d50774aaa616681dfe28b23b3edc355
I20260924 07:13:48.349772 21870 leader_election.cc:302] T 00000000000000000000000000000000 P cd84d1d369b64a948294acb15c960ba7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd84d1d369b64a948294acb15c960ba7; no voters:
I20260924 07:13:48.349925 21870 leader_election.cc:288] T 00000000000000000000000000000000 P cd84d1d369b64a948294acb15c960ba7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:13:48.349978 21873 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd84d1d369b64a948294acb15c960ba7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:13:48.350163 21873 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd84d1d369b64a948294acb15c960ba7 [term 1 LEADER]: Becoming Leader. State: Replica: cd84d1d369b64a948294acb15c960ba7, State: Running, Role: LEADER
I20260924 07:13:48.350183 21870 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd84d1d369b64a948294acb15c960ba7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:13:48.350317 21873 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd84d1d369b64a948294acb15c960ba7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd84d1d369b64a948294acb15c960ba7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } }
I20260924 07:13:48.350739 21875 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd84d1d369b64a948294acb15c960ba7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd84d1d369b64a948294acb15c960ba7. Latest consensus state: current_term: 1 leader_uuid: "cd84d1d369b64a948294acb15c960ba7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd84d1d369b64a948294acb15c960ba7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } } }
I20260924 07:13:48.350744 21874 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd84d1d369b64a948294acb15c960ba7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd84d1d369b64a948294acb15c960ba7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd84d1d369b64a948294acb15c960ba7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43105 } } }
I20260924 07:13:48.350821 21875 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd84d1d369b64a948294acb15c960ba7 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:48.350836 21874 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd84d1d369b64a948294acb15c960ba7 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:48.351303 21879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:13:48.352022 21879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:13:48.352229 21877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.350318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.68:44889 (local address 127.0.0.1:43105)
0924 07:13:48.350541 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:48.350550 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:48.350605 (+ 55us) server_negotiation.cc:407] Connection header received
0924 07:13:48.350795 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:48.350803 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:48.350834 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:48.350974 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:48.351621 (+ 647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:48.351625 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:48.351625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:13:48.351630 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:48.351676 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:13:48.351692 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:48.351738 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:48.351901 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 07:13:48.351954 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":66,"threads_started":1}
I20260924 07:13:48.353235 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:13:48.355330 21879 catalog_manager.cc:1347] Generated new cluster ID: a5fe3db357bd4f318168ac3f64d8bb51
I20260924 07:13:48.355398 21879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:13:48.355835 21877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:48.354748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.68:59579 (local address 127.0.0.1:43105)
0924 07:13:48.354791 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:48.354793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:48.354941 (+ 148us) server_negotiation.cc:407] Connection header received
0924 07:13:48.355000 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:48.355002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:48.355007 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:48.355078 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:48.355635 (+ 557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:48.355636 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:48.355637 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:13:48.355640 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:48.355652 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:13:48.355656 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:48.355708 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:48.355771 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 07:13:48.355783 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 07:13:48.363162 21879 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:13:48.363304 21879 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:13:48.365962 21879 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd84d1d369b64a948294acb15c960ba7: Generated new TSK 0
I20260924 07:13:48.366125 21879 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:13:48.368386 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 21802
2026-09-24T07:13:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1295 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:13:48 dist-test-slave-14ww krb5kdc[21901](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:13:48 dist-test-slave-14ww krb5kdc[21901](info): set up 2 sockets
Sep 24 07:13:48 dist-test-slave-14ww krb5kdc[21901](info): commencing operation
krb5kdc: starting...
W20260924 07:13:49.396808 392 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:13:49 dist-test-slave-14ww krb5kdc[21901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:13:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:13:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:13:49.511821 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44617
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:49.590174 21914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:49.590396 21914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:49.590433 21914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:49.591547 21914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:13:49.591612 21914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:49.591636 21914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:49.591657 21914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:13:49.591677 21914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:13:49.593124 21914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44617
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:49.593386 21914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:49.593606 21914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:49.598488 21919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:49.598603 21922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:49.598515 21920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:49.598512 21914 server_base.cc:1034] running on GCE node
I20260924 07:13:49.598886 21914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:49.599175 21914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:49.600345 21914 hybrid_clock.cc:648] HybridClock initialized: now 1790234029600308 us; error 47 us; skew 500 ppm
Sep 24 07:13:49 dist-test-slave-14ww krb5kdc[21901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:49.602638 21914 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:13:49.603123 21914 webserver.cc:460] Webserver started at http://127.0.98.62:41643/ using document root <none> and password file <none>
I20260924 07:13:49.603314 21914 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:49.603353 21914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:49.603433 21914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:49.604373 21914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "90f1139a668844a9a306b993b7a54c5c"
format_stamp: "Formatted at 2026-09-24 07:13:49 on dist-test-slave-14ww"
server_key: "1cee44604626ad9411742797c855636e"
server_key_iv: "11cbd948665d0f2c67f89b72517f74de"
server_key_version: "encryptionkey@0"
I20260924 07:13:49.604658 21914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "90f1139a668844a9a306b993b7a54c5c"
format_stamp: "Formatted at 2026-09-24 07:13:49 on dist-test-slave-14ww"
server_key: "1cee44604626ad9411742797c855636e"
server_key_iv: "11cbd948665d0f2c67f89b72517f74de"
server_key_version: "encryptionkey@0"
I20260924 07:13:49.605912 21914 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:13:49.606673 21929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:49.606873 21914 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:13:49.606956 21914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "90f1139a668844a9a306b993b7a54c5c"
format_stamp: "Formatted at 2026-09-24 07:13:49 on dist-test-slave-14ww"
server_key: "1cee44604626ad9411742797c855636e"
server_key_iv: "11cbd948665d0f2c67f89b72517f74de"
server_key_version: "encryptionkey@0"
I20260924 07:13:49.607007 21914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:49.627943 21914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:49.629961 21914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:49.630074 21914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:49.634495 21914 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44617
I20260924 07:13:49.634524 21981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44617 every 8 connection(s)
I20260924 07:13:49.634899 21914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:13:49.635793 21982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:49.636018 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 21914
I20260924 07:13:49.636152 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:13:49.636353 392 external_mini_cluster.cc:1428] Setting key 36c46e4a6c0c87be3b5e0dbde27f4944
I20260924 07:13:49.639186 21982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90f1139a668844a9a306b993b7a54c5c: Bootstrap starting.
I20260924 07:13:49.639840 21982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90f1139a668844a9a306b993b7a54c5c: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:49.640136 21982 log.cc:824] T 00000000000000000000000000000000 P 90f1139a668844a9a306b993b7a54c5c: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:49.640879 21982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90f1139a668844a9a306b993b7a54c5c: No bootstrap required, opened a new log
Sep 24 07:13:49 dist-test-slave-14ww krb5kdc[21901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:13:49.643461 21982 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90f1139a668844a9a306b993b7a54c5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90f1139a668844a9a306b993b7a54c5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44617 } }
I20260924 07:13:49.643610 21982 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90f1139a668844a9a306b993b7a54c5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:49.643646 21982 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90f1139a668844a9a306b993b7a54c5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90f1139a668844a9a306b993b7a54c5c, State: Initialized, Role: FOLLOWER
I20260924 07:13:49.643779 21982 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90f1139a668844a9a306b993b7a54c5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90f1139a668844a9a306b993b7a54c5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44617 } }
I20260924 07:13:49.643860 21982 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90f1139a668844a9a306b993b7a54c5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:13:49.643904 21982 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90f1139a668844a9a306b993b7a54c5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:13:49.643951 21982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90f1139a668844a9a306b993b7a54c5c [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:49.644642 21982 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90f1139a668844a9a306b993b7a54c5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90f1139a668844a9a306b993b7a54c5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44617 } }
I20260924 07:13:49.644779 21982 leader_election.cc:302] T 00000000000000000000000000000000 P 90f1139a668844a9a306b993b7a54c5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90f1139a668844a9a306b993b7a54c5c; no voters:
I20260924 07:13:49.644941 21982 leader_election.cc:288] T 00000000000000000000000000000000 P 90f1139a668844a9a306b993b7a54c5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:13:49.644991 21987 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90f1139a668844a9a306b993b7a54c5c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:13:49.645139 21987 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90f1139a668844a9a306b993b7a54c5c [term 1 LEADER]: Becoming Leader. State: Replica: 90f1139a668844a9a306b993b7a54c5c, State: Running, Role: LEADER
I20260924 07:13:49.645227 21982 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90f1139a668844a9a306b993b7a54c5c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:13:49.645324 21987 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90f1139a668844a9a306b993b7a54c5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90f1139a668844a9a306b993b7a54c5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44617 } }
I20260924 07:13:49.645720 21987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90f1139a668844a9a306b993b7a54c5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90f1139a668844a9a306b993b7a54c5c. Latest consensus state: current_term: 1 leader_uuid: "90f1139a668844a9a306b993b7a54c5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90f1139a668844a9a306b993b7a54c5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44617 } } }
I20260924 07:13:49.645725 21989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90f1139a668844a9a306b993b7a54c5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90f1139a668844a9a306b993b7a54c5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90f1139a668844a9a306b993b7a54c5c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44617 } } }
I20260924 07:13:49.645896 21989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90f1139a668844a9a306b993b7a54c5c [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:49.646081 21987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90f1139a668844a9a306b993b7a54c5c [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:49.646459 21996 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:13:49.647028 21996 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:13:49.647759 21985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:49.636895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35463 (local address 127.0.98.62:44617)
0924 07:13:49.637077 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:49.637094 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:49.637127 (+ 33us) server_negotiation.cc:407] Connection header received
0924 07:13:49.637575 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:49.637581 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:49.637632 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:49.637753 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:49.638539 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:49.639555 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:49.640599 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:49.641102 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:49.641195 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:49.643792 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:49.643797 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:49.643797 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:49.643809 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:49.645873 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:49.646347 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:49.646350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:49.646351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:49.646448 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:49.646750 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:49.646751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:49.646752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:49.647045 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:49.647174 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:49.647462 (+ 288us) server_negotiation.cc:299] Negotiation successful
0924 07:13:49.647563 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":100,"threads_started":1}
I20260924 07:13:49.648674 21996 catalog_manager.cc:1347] Generated new cluster ID: 01b21d52200c4f2fa1d19d06ffedefe1
I20260924 07:13:49.648748 21996 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:13:49.653075 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:13:49.660094 21996 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:13:49.660768 21996 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:13:49.661562 21985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:49.654444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40591 (local address 127.0.98.62:44617)
0924 07:13:49.654486 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:49.654488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:49.654566 (+ 78us) server_negotiation.cc:407] Connection header received
0924 07:13:49.654598 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:49.654599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:49.654609 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:49.654676 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:49.655238 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:49.655996 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:49.656981 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:49.657347 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:49.657418 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:49.658745 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:49.658747 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:49.658747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:49.658754 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:49.660344 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:49.660798 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:49.660799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:49.660800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:49.660858 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:49.661082 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:49.661083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:49.661083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:49.661205 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:49.661315 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:49.661415 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 07:13:49.661477 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 07:13:49.663115 21996 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90f1139a668844a9a306b993b7a54c5c: Generated new TSK 0
I20260924 07:13:49.663271 21996 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:13:49.671607 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 21914
W20260924 07:13:49.675722 21899 connection.cc:535] client connection to 127.0.98.62:44617 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:44617): Connection reset by peer (error 104)
2026-09-24T07:13:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1299 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:13:49 dist-test-slave-14ww krb5kdc[22013](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:13:49 dist-test-slave-14ww krb5kdc[22013](info): set up 2 sockets
Sep 24 07:13:49 dist-test-slave-14ww krb5kdc[22013](info): commencing operation
krb5kdc: starting...
W20260924 07:13:50.708904 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:13:50 dist-test-slave-14ww krb5kdc[22013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:13:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:13:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:13:50.826109 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36135
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:50.906337 22026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:50.906565 22026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:50.906610 22026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:50.907737 22026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:13:50.907806 22026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:50.907831 22026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:50.907858 22026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:13:50.907891 22026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:13:50.909363 22026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36135
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:50.909636 22026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:50.909858 22026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:50.914732 22032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:50.914732 22034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:50.915019 22026 server_base.cc:1034] running on GCE node
W20260924 07:13:50.914793 22031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:50.915371 22026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:50.915663 22026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:50.916851 22026 hybrid_clock.cc:648] HybridClock initialized: now 1790234030916838 us; error 51 us; skew 500 ppm
Sep 24 07:13:50 dist-test-slave-14ww krb5kdc[22013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:50.919763 22026 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:13:50.920197 22026 webserver.cc:460] Webserver started at http://127.0.98.62:41045/ using document root <none> and password file <none>
I20260924 07:13:50.920441 22026 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:50.920488 22026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:50.920619 22026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:50.921674 22026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "62de6f45b28c4b658254b3a5d0e5050f"
format_stamp: "Formatted at 2026-09-24 07:13:50 on dist-test-slave-14ww"
server_key: "50facdfde91a36e4a99ad2820b71c149"
server_key_iv: "90ea91c4f22bec7db9ce8b3d7ddbe08f"
server_key_version: "encryptionkey@0"
I20260924 07:13:50.922022 22026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "62de6f45b28c4b658254b3a5d0e5050f"
format_stamp: "Formatted at 2026-09-24 07:13:50 on dist-test-slave-14ww"
server_key: "50facdfde91a36e4a99ad2820b71c149"
server_key_iv: "90ea91c4f22bec7db9ce8b3d7ddbe08f"
server_key_version: "encryptionkey@0"
I20260924 07:13:50.923332 22026 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:13:50.924069 22041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:50.924232 22026 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:13:50.924332 22026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "62de6f45b28c4b658254b3a5d0e5050f"
format_stamp: "Formatted at 2026-09-24 07:13:50 on dist-test-slave-14ww"
server_key: "50facdfde91a36e4a99ad2820b71c149"
server_key_iv: "90ea91c4f22bec7db9ce8b3d7ddbe08f"
server_key_version: "encryptionkey@0"
I20260924 07:13:50.924410 22026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:50.942996 22026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:50.945436 22026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:50.945595 22026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:50.949672 22026 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36135
I20260924 07:13:50.949709 22093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36135 every 8 connection(s)
I20260924 07:13:50.950059 22026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:13:50.950701 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 22026
I20260924 07:13:50.950829 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:13:50.951025 392 external_mini_cluster.cc:1428] Setting key 7ad0e7d7c3301cce83b0f8a8215beb63
I20260924 07:13:50.951184 22094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:50.954038 22094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62de6f45b28c4b658254b3a5d0e5050f: Bootstrap starting.
I20260924 07:13:50.954730 22094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62de6f45b28c4b658254b3a5d0e5050f: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:50.955001 22094 log.cc:824] T 00000000000000000000000000000000 P 62de6f45b28c4b658254b3a5d0e5050f: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:50.955811 22094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62de6f45b28c4b658254b3a5d0e5050f: No bootstrap required, opened a new log
I20260924 07:13:50.956549 22097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:50.951558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35707 (local address 127.0.98.62:36135)
0924 07:13:50.951779 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:50.951786 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:50.951810 (+ 24us) server_negotiation.cc:407] Connection header received
0924 07:13:50.952260 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:50.952266 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:50.952311 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:50.952455 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:50.952687 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:50.953728 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:50.954562 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:50.955097 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:50.955208 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:50.955901 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:50.955906 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:50.955906 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:13:50.956077 (+ 171us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:50.956106 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:13:50.956119 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:50.956184 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:50.956307 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 07:13:50.956348 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":106,"threads_started":1}
I20260924 07:13:50.957631 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:13:50.958209 22094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62de6f45b28c4b658254b3a5d0e5050f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62de6f45b28c4b658254b3a5d0e5050f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36135 } }
I20260924 07:13:50.958389 22094 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62de6f45b28c4b658254b3a5d0e5050f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:50.958429 22094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62de6f45b28c4b658254b3a5d0e5050f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62de6f45b28c4b658254b3a5d0e5050f, State: Initialized, Role: FOLLOWER
I20260924 07:13:50.958546 22094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62de6f45b28c4b658254b3a5d0e5050f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62de6f45b28c4b658254b3a5d0e5050f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36135 } }
I20260924 07:13:50.958621 22094 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62de6f45b28c4b658254b3a5d0e5050f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:13:50.958658 22094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62de6f45b28c4b658254b3a5d0e5050f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:13:50.958783 22094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62de6f45b28c4b658254b3a5d0e5050f [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:50.959417 22094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62de6f45b28c4b658254b3a5d0e5050f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62de6f45b28c4b658254b3a5d0e5050f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36135 } }
I20260924 07:13:50.959533 22094 leader_election.cc:302] T 00000000000000000000000000000000 P 62de6f45b28c4b658254b3a5d0e5050f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62de6f45b28c4b658254b3a5d0e5050f; no voters:
I20260924 07:13:50.959694 22094 leader_election.cc:288] T 00000000000000000000000000000000 P 62de6f45b28c4b658254b3a5d0e5050f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:13:50.959753 22104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62de6f45b28c4b658254b3a5d0e5050f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:13:50.959920 22094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62de6f45b28c4b658254b3a5d0e5050f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:13:50.959933 22104 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62de6f45b28c4b658254b3a5d0e5050f [term 1 LEADER]: Becoming Leader. State: Replica: 62de6f45b28c4b658254b3a5d0e5050f, State: Running, Role: LEADER
I20260924 07:13:50.960110 22104 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62de6f45b28c4b658254b3a5d0e5050f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62de6f45b28c4b658254b3a5d0e5050f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36135 } }
I20260924 07:13:50.960439 22105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62de6f45b28c4b658254b3a5d0e5050f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62de6f45b28c4b658254b3a5d0e5050f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62de6f45b28c4b658254b3a5d0e5050f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36135 } } }
I20260924 07:13:50.960564 22105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62de6f45b28c4b658254b3a5d0e5050f [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:50.961642 22106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62de6f45b28c4b658254b3a5d0e5050f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62de6f45b28c4b658254b3a5d0e5050f. Latest consensus state: current_term: 1 leader_uuid: "62de6f45b28c4b658254b3a5d0e5050f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62de6f45b28c4b658254b3a5d0e5050f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36135 } } }
I20260924 07:13:50.961782 22106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62de6f45b28c4b658254b3a5d0e5050f [sys.catalog]: This master's current role is: LEADER
W20260924 07:13:50.961961 22120 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 62de6f45b28c4b658254b3a5d0e5050f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:13:50.962018 22120 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:13:50.962116 22111 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:13:50.962303 22111 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:13:50.962729 22097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:50.959135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56281 (local address 127.0.98.62:36135)
0924 07:13:50.959176 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:50.959178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:50.959300 (+ 122us) server_negotiation.cc:407] Connection header received
0924 07:13:50.959369 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:50.959370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:50.959379 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:50.959433 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:50.959593 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:50.960278 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:50.961348 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:50.961674 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:50.961745 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:50.962438 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:50.962439 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:50.962439 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:13:50.962443 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:50.962456 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:13:50.962461 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:50.962515 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:50.962650 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 07:13:50.962660 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 07:13:50.963881 22111 catalog_manager.cc:1347] Generated new cluster ID: 5588b73e2f524f74932e0ccc226da9ef
I20260924 07:13:50.963936 22111 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:13:50.976990 22111 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:13:50.977694 22111 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:13:50.993037 22111 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62de6f45b28c4b658254b3a5d0e5050f: Generated new TSK 0
I20260924 07:13:50.993261 22111 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:13:51.001325 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 22026
W20260924 07:13:51.005699 22011 connection.cc:535] client connection to 127.0.98.62:36135 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:36135): Connection reset by peer (error 104)
2026-09-24T07:13:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1330 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:13:51 dist-test-slave-14ww krb5kdc[22125](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:13:51 dist-test-slave-14ww krb5kdc[22125](info): set up 2 sockets
Sep 24 07:13:51 dist-test-slave-14ww krb5kdc[22125](info): commencing operation
krb5kdc: starting...
W20260924 07:13:52.040894 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:13:52 dist-test-slave-14ww krb5kdc[22125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:13:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:13:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:13:52.155993 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34883
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:52.235306 22138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:52.235498 22138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:52.235522 22138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:52.236673 22138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:13:52.236716 22138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:52.236752 22138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:52.236765 22138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:13:52.236829 22138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:13:52.238276 22138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34883
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:52.238523 22138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:52.238746 22138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:52.243904 22146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:52.243981 22144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:52.243908 22143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:52.244069 22138 server_base.cc:1034] running on GCE node
I20260924 07:13:52.244359 22138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:52.244661 22138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:52.245806 22138 hybrid_clock.cc:648] HybridClock initialized: now 1790234032245769 us; error 50 us; skew 500 ppm
Sep 24 07:13:52 dist-test-slave-14ww krb5kdc[22125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:52.248157 22138 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:13:52.248597 22138 webserver.cc:460] Webserver started at http://127.0.98.62:37829/ using document root <none> and password file <none>
I20260924 07:13:52.248836 22138 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:52.248883 22138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:52.248971 22138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:52.249943 22138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "64aa5ceaf0564735a3580a8588b7fbff"
format_stamp: "Formatted at 2026-09-24 07:13:52 on dist-test-slave-14ww"
server_key: "eb3e8ce10ec4eae23ae0314864f2b2c1"
server_key_iv: "144c321d23ba9a5841a9abc16dd17382"
server_key_version: "encryptionkey@0"
I20260924 07:13:52.250289 22138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "64aa5ceaf0564735a3580a8588b7fbff"
format_stamp: "Formatted at 2026-09-24 07:13:52 on dist-test-slave-14ww"
server_key: "eb3e8ce10ec4eae23ae0314864f2b2c1"
server_key_iv: "144c321d23ba9a5841a9abc16dd17382"
server_key_version: "encryptionkey@0"
I20260924 07:13:52.251610 22138 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:13:52.252331 22153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:52.252503 22138 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:13:52.252596 22138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "64aa5ceaf0564735a3580a8588b7fbff"
format_stamp: "Formatted at 2026-09-24 07:13:52 on dist-test-slave-14ww"
server_key: "eb3e8ce10ec4eae23ae0314864f2b2c1"
server_key_iv: "144c321d23ba9a5841a9abc16dd17382"
server_key_version: "encryptionkey@0"
I20260924 07:13:52.252672 22138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:52.255811 22138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:52.257580 22138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:52.257726 22138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:52.261632 22138 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34883
I20260924 07:13:52.261662 22205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34883 every 8 connection(s)
I20260924 07:13:52.261967 22138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:13:52.262787 22206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:52.264917 22206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64aa5ceaf0564735a3580a8588b7fbff: Bootstrap starting.
I20260924 07:13:52.265435 22206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64aa5ceaf0564735a3580a8588b7fbff: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:52.265668 22206 log.cc:824] T 00000000000000000000000000000000 P 64aa5ceaf0564735a3580a8588b7fbff: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:52.266331 22206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64aa5ceaf0564735a3580a8588b7fbff: No bootstrap required, opened a new log
I20260924 07:13:52.267848 22206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64aa5ceaf0564735a3580a8588b7fbff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64aa5ceaf0564735a3580a8588b7fbff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34883 } }
I20260924 07:13:52.267975 22206 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64aa5ceaf0564735a3580a8588b7fbff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:52.267997 22206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64aa5ceaf0564735a3580a8588b7fbff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64aa5ceaf0564735a3580a8588b7fbff, State: Initialized, Role: FOLLOWER
I20260924 07:13:52.268136 22206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64aa5ceaf0564735a3580a8588b7fbff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64aa5ceaf0564735a3580a8588b7fbff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34883 } }
I20260924 07:13:52.268215 22206 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64aa5ceaf0564735a3580a8588b7fbff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:13:52.268237 22206 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64aa5ceaf0564735a3580a8588b7fbff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:13:52.268265 22206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64aa5ceaf0564735a3580a8588b7fbff [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:52.268879 22206 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64aa5ceaf0564735a3580a8588b7fbff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64aa5ceaf0564735a3580a8588b7fbff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34883 } }
I20260924 07:13:52.268987 22206 leader_election.cc:302] T 00000000000000000000000000000000 P 64aa5ceaf0564735a3580a8588b7fbff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64aa5ceaf0564735a3580a8588b7fbff; no voters:
I20260924 07:13:52.269162 22206 leader_election.cc:288] T 00000000000000000000000000000000 P 64aa5ceaf0564735a3580a8588b7fbff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:13:52.269227 22209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64aa5ceaf0564735a3580a8588b7fbff [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:13:52.269407 22206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64aa5ceaf0564735a3580a8588b7fbff [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:13:52.269418 22209 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64aa5ceaf0564735a3580a8588b7fbff [term 1 LEADER]: Becoming Leader. State: Replica: 64aa5ceaf0564735a3580a8588b7fbff, State: Running, Role: LEADER
I20260924 07:13:52.269610 22209 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64aa5ceaf0564735a3580a8588b7fbff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64aa5ceaf0564735a3580a8588b7fbff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34883 } }
I20260924 07:13:52.270277 22211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64aa5ceaf0564735a3580a8588b7fbff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64aa5ceaf0564735a3580a8588b7fbff. Latest consensus state: current_term: 1 leader_uuid: "64aa5ceaf0564735a3580a8588b7fbff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64aa5ceaf0564735a3580a8588b7fbff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34883 } } }
I20260924 07:13:52.270354 22211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64aa5ceaf0564735a3580a8588b7fbff [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:52.270409 22210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64aa5ceaf0564735a3580a8588b7fbff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64aa5ceaf0564735a3580a8588b7fbff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64aa5ceaf0564735a3580a8588b7fbff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34883 } } }
I20260924 07:13:52.270484 22210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64aa5ceaf0564735a3580a8588b7fbff [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:52.270524 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 22138
I20260924 07:13:52.270642 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:13:52.270852 392 external_mini_cluster.cc:1428] Setting key c114a6cb24eec0c810ca1b624ed898eb
I20260924 07:13:52.271450 22218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:13:52.271958 22218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:13:52.273141 22225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:52.271399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49261 (local address 127.0.98.62:34883)
0924 07:13:52.271585 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:52.271596 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:52.271626 (+ 30us) server_negotiation.cc:407] Connection header received
0924 07:13:52.271739 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:52.271746 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:52.271774 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:52.271917 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:52.272630 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:52.272634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:52.272634 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:13:52.272639 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:52.272679 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:13:52.272695 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:52.272808 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:52.272877 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 07:13:52.272927 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":72,"threads_started":1}
I20260924 07:13:52.273830 22218 catalog_manager.cc:1347] Generated new cluster ID: 0994965e548d4f38afeba180661c2bac
I20260924 07:13:52.273880 22218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:13:52.277021 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:13:52.282615 22225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:52.281095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35001 (local address 127.0.98.62:34883)
0924 07:13:52.281148 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:52.281150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:52.281685 (+ 535us) server_negotiation.cc:407] Connection header received
0924 07:13:52.281757 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:52.281758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:52.281763 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:52.281818 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:52.282406 (+ 588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:52.282407 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:52.282408 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:13:52.282412 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:52.282426 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:13:52.282430 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:52.282476 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:52.282538 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 07:13:52.282552 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 07:13:52.295388 22218 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:13:52.295558 22218 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:13:52.299299 22218 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64aa5ceaf0564735a3580a8588b7fbff: Generated new TSK 0
I20260924 07:13:52.299464 22218 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:13:52.302042 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 22138
2026-09-24T07:13:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1300 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:13:52 dist-test-slave-14ww krb5kdc[22237](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:13:52 dist-test-slave-14ww krb5kdc[22237](info): set up 2 sockets
Sep 24 07:13:52 dist-test-slave-14ww krb5kdc[22237](info): commencing operation
krb5kdc: starting...
W20260924 07:13:53.332868 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:13:53 dist-test-slave-14ww krb5kdc[22237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:13:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:13:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:13:53.452548 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37987
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:53.531477 22250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:53.531699 22250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:53.531738 22250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:53.532946 22250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:13:53.533007 22250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:53.533033 22250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:53.533082 22250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:13:53.533113 22250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:13:53.534564 22250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:53.534873 22250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:53.535102 22250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:53.539901 22256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:53.540009 22255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:53.540045 22250 server_base.cc:1034] running on GCE node
W20260924 07:13:53.540191 22258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:53.540453 22250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:53.540858 22250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:53.542025 22250 hybrid_clock.cc:648] HybridClock initialized: now 1790234033542010 us; error 44 us; skew 500 ppm
Sep 24 07:13:53 dist-test-slave-14ww krb5kdc[22237](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:53.560658 22250 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:13:53.561228 22250 webserver.cc:460] Webserver started at http://127.0.98.62:42719/ using document root <none> and password file <none>
I20260924 07:13:53.561488 22250 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:53.561561 22250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:53.561729 22250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:53.562840 22250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "a2c0803997224777a297700cf0be3e26"
format_stamp: "Formatted at 2026-09-24 07:13:53 on dist-test-slave-14ww"
server_key: "18d7009841bdf8e2ce38c0dbf705b8aa"
server_key_iv: "de90149a2ace181b4ddc958d69558bd8"
server_key_version: "encryptionkey@0"
I20260924 07:13:53.563217 22250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "a2c0803997224777a297700cf0be3e26"
format_stamp: "Formatted at 2026-09-24 07:13:53 on dist-test-slave-14ww"
server_key: "18d7009841bdf8e2ce38c0dbf705b8aa"
server_key_iv: "de90149a2ace181b4ddc958d69558bd8"
server_key_version: "encryptionkey@0"
I20260924 07:13:53.564582 22250 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:13:53.565455 22265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:53.565644 22250 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:13:53.565729 22250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "a2c0803997224777a297700cf0be3e26"
format_stamp: "Formatted at 2026-09-24 07:13:53 on dist-test-slave-14ww"
server_key: "18d7009841bdf8e2ce38c0dbf705b8aa"
server_key_iv: "de90149a2ace181b4ddc958d69558bd8"
server_key_version: "encryptionkey@0"
I20260924 07:13:53.565821 22250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:53.586155 22250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:53.588604 22250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:53.588793 22250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:53.592875 22250 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37987
I20260924 07:13:53.592947 22317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37987 every 8 connection(s)
I20260924 07:13:53.593315 22250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:13:53.594175 22318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:53.596405 22318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2c0803997224777a297700cf0be3e26: Bootstrap starting.
I20260924 07:13:53.596884 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 22250
I20260924 07:13:53.596995 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:13:53.597152 22318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2c0803997224777a297700cf0be3e26: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:53.597205 392 external_mini_cluster.cc:1428] Setting key 32fd2ab26b97d2c8e412eaf1dd2f9280
I20260924 07:13:53.597479 22318 log.cc:824] T 00000000000000000000000000000000 P a2c0803997224777a297700cf0be3e26: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:53.598258 22318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2c0803997224777a297700cf0be3e26: No bootstrap required, opened a new log
I20260924 07:13:53.600654 22318 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2c0803997224777a297700cf0be3e26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2c0803997224777a297700cf0be3e26" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37987 } }
I20260924 07:13:53.600863 22318 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2c0803997224777a297700cf0be3e26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:53.600931 22318 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2c0803997224777a297700cf0be3e26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c0803997224777a297700cf0be3e26, State: Initialized, Role: FOLLOWER
I20260924 07:13:53.601097 22318 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2c0803997224777a297700cf0be3e26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2c0803997224777a297700cf0be3e26" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37987 } }
I20260924 07:13:53.601176 22318 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2c0803997224777a297700cf0be3e26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:13:53.601200 22318 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2c0803997224777a297700cf0be3e26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:13:53.601266 22318 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2c0803997224777a297700cf0be3e26 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:53.601835 22318 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2c0803997224777a297700cf0be3e26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2c0803997224777a297700cf0be3e26" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37987 } }
I20260924 07:13:53.601938 22318 leader_election.cc:302] T 00000000000000000000000000000000 P a2c0803997224777a297700cf0be3e26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2c0803997224777a297700cf0be3e26; no voters:
I20260924 07:13:53.602084 22318 leader_election.cc:288] T 00000000000000000000000000000000 P a2c0803997224777a297700cf0be3e26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:13:53.602150 22323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2c0803997224777a297700cf0be3e26 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:13:53.602311 22318 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2c0803997224777a297700cf0be3e26 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:13:53.602339 22323 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2c0803997224777a297700cf0be3e26 [term 1 LEADER]: Becoming Leader. State: Replica: a2c0803997224777a297700cf0be3e26, State: Running, Role: LEADER
I20260924 07:13:53.602559 22323 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2c0803997224777a297700cf0be3e26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2c0803997224777a297700cf0be3e26" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37987 } }
I20260924 07:13:53.602917 22324 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2c0803997224777a297700cf0be3e26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2c0803997224777a297700cf0be3e26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2c0803997224777a297700cf0be3e26" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37987 } } }
I20260924 07:13:53.603001 22324 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2c0803997224777a297700cf0be3e26 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:53.603159 22325 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2c0803997224777a297700cf0be3e26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2c0803997224777a297700cf0be3e26. Latest consensus state: current_term: 1 leader_uuid: "a2c0803997224777a297700cf0be3e26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2c0803997224777a297700cf0be3e26" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37987 } } }
I20260924 07:13:53.603293 22325 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2c0803997224777a297700cf0be3e26 [sys.catalog]: This master's current role is: LEADER
W20260924 07:13:53.604231 22338 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a2c0803997224777a297700cf0be3e26: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:13:53.604406 22338 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:13:53.604816 22332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 24 07:13:53 dist-test-slave-14ww krb5kdc[22237](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:13:53.605043 22332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:13:53.606590 22332 catalog_manager.cc:1347] Generated new cluster ID: 7133a6e9df234d828c9df962a52407d4
I20260924 07:13:53.606669 22332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:13:53.609071 22322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:53.597676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34269 (local address 127.0.98.62:37987)
0924 07:13:53.597840 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:53.597850 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:53.597885 (+ 35us) server_negotiation.cc:407] Connection header received
0924 07:13:53.598487 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:53.598495 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:53.598556 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:53.598715 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:53.599337 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:53.600807 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:53.601762 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:53.602392 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:53.602887 (+ 495us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:53.605424 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:53.605428 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:53.605428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:53.605440 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:53.607430 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:53.607821 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:53.607824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:53.607825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:53.607937 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:53.608141 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:53.608142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:53.608143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:53.608445 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:53.608600 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:53.608679 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 07:13:53.608843 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":77,"threads_started":1}
I20260924 07:13:53.609923 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:13:53.623863 22322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:53.613038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54733 (local address 127.0.98.62:37987)
0924 07:13:53.613081 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:53.613082 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:53.616920 (+ 3838us) server_negotiation.cc:407] Connection header received
0924 07:13:53.616952 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:53.616953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:53.616964 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:53.617029 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:53.617627 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:53.618385 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:53.619346 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:53.619694 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:53.619767 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:53.621176 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:53.621178 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:53.621178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:53.621185 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:53.622757 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:53.623113 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:53.623115 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:53.623115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:53.623172 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:53.623366 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:53.623367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:53.623367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:53.623496 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:53.623591 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:53.623689 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 07:13:53.623761 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 07:13:53.628440 22332 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:13:53.629012 22332 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:13:53.631184 22332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2c0803997224777a297700cf0be3e26: Generated new TSK 0
I20260924 07:13:53.631371 22332 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:13:53.635317 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 22250
2026-09-24T07:13:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:13:53 dist-test-slave-14ww krb5kdc[22349](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:13:53 dist-test-slave-14ww krb5kdc[22349](info): set up 2 sockets
Sep 24 07:13:53 dist-test-slave-14ww krb5kdc[22349](info): commencing operation
krb5kdc: starting...
W20260924 07:13:54.681399 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:13:54 dist-test-slave-14ww krb5kdc[22349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:13:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:13:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:13:54.799031 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43445
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:54.878475 22362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:54.878680 22362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:54.878703 22362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:54.879904 22362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:13:54.879947 22362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:54.879962 22362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:54.879976 22362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:13:54.879993 22362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:13:54.881423 22362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43445
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:54.881614 22362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:54.881781 22362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:54.886825 22368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:54.886801 22370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:54.886902 22362 server_base.cc:1034] running on GCE node
W20260924 07:13:54.886858 22367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:54.887193 22362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:54.887498 22362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:54.888862 22362 hybrid_clock.cc:648] HybridClock initialized: now 1790234034888818 us; error 41 us; skew 500 ppm
Sep 24 07:13:54 dist-test-slave-14ww krb5kdc[22349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:54.892146 22362 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:13:54.892616 22362 webserver.cc:460] Webserver started at http://127.0.98.62:33859/ using document root <none> and password file <none>
I20260924 07:13:54.892881 22362 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:54.892931 22362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:54.893061 22362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:54.894140 22362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "cd387d5ae0ca419f8ec7f545d58b7615"
format_stamp: "Formatted at 2026-09-24 07:13:54 on dist-test-slave-14ww"
server_key: "6c1ab9c71cf5312ca5e0da7db79094f4"
server_key_iv: "ec1ad7cb7dbf43e81ef2a3c706255665"
server_key_version: "encryptionkey@0"
I20260924 07:13:54.894501 22362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "cd387d5ae0ca419f8ec7f545d58b7615"
format_stamp: "Formatted at 2026-09-24 07:13:54 on dist-test-slave-14ww"
server_key: "6c1ab9c71cf5312ca5e0da7db79094f4"
server_key_iv: "ec1ad7cb7dbf43e81ef2a3c706255665"
server_key_version: "encryptionkey@0"
I20260924 07:13:54.895807 22362 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:13:54.896560 22377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:54.896819 22362 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:13:54.896914 22362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "cd387d5ae0ca419f8ec7f545d58b7615"
format_stamp: "Formatted at 2026-09-24 07:13:54 on dist-test-slave-14ww"
server_key: "6c1ab9c71cf5312ca5e0da7db79094f4"
server_key_iv: "ec1ad7cb7dbf43e81ef2a3c706255665"
server_key_version: "encryptionkey@0"
I20260924 07:13:54.896988 22362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:54.947142 22362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:54.949311 22362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:54.949450 22362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:54.953722 22362 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43445
I20260924 07:13:54.953781 22429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43445 every 8 connection(s)
I20260924 07:13:54.954161 22362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:13:54.955216 22430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:54.957403 22430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd387d5ae0ca419f8ec7f545d58b7615: Bootstrap starting.
I20260924 07:13:54.957914 22430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd387d5ae0ca419f8ec7f545d58b7615: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:54.958148 22430 log.cc:824] T 00000000000000000000000000000000 P cd387d5ae0ca419f8ec7f545d58b7615: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:54.959043 22430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd387d5ae0ca419f8ec7f545d58b7615: No bootstrap required, opened a new log
I20260924 07:13:54.960582 22430 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd387d5ae0ca419f8ec7f545d58b7615 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd387d5ae0ca419f8ec7f545d58b7615" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43445 } }
I20260924 07:13:54.960716 22430 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd387d5ae0ca419f8ec7f545d58b7615 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:54.960819 22430 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd387d5ae0ca419f8ec7f545d58b7615 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd387d5ae0ca419f8ec7f545d58b7615, State: Initialized, Role: FOLLOWER
I20260924 07:13:54.960944 22430 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd387d5ae0ca419f8ec7f545d58b7615 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd387d5ae0ca419f8ec7f545d58b7615" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43445 } }
I20260924 07:13:54.961010 22430 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd387d5ae0ca419f8ec7f545d58b7615 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:13:54.961031 22430 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd387d5ae0ca419f8ec7f545d58b7615 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:13:54.961058 22430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd387d5ae0ca419f8ec7f545d58b7615 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:54.961519 22430 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd387d5ae0ca419f8ec7f545d58b7615 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd387d5ae0ca419f8ec7f545d58b7615" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43445 } }
I20260924 07:13:54.961599 22430 leader_election.cc:302] T 00000000000000000000000000000000 P cd387d5ae0ca419f8ec7f545d58b7615 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd387d5ae0ca419f8ec7f545d58b7615; no voters:
I20260924 07:13:54.961715 22430 leader_election.cc:288] T 00000000000000000000000000000000 P cd387d5ae0ca419f8ec7f545d58b7615 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:13:54.961767 22433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd387d5ae0ca419f8ec7f545d58b7615 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:13:54.961925 22433 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd387d5ae0ca419f8ec7f545d58b7615 [term 1 LEADER]: Becoming Leader. State: Replica: cd387d5ae0ca419f8ec7f545d58b7615, State: Running, Role: LEADER
I20260924 07:13:54.961972 22430 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd387d5ae0ca419f8ec7f545d58b7615 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:13:54.962059 22433 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd387d5ae0ca419f8ec7f545d58b7615 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd387d5ae0ca419f8ec7f545d58b7615" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43445 } }
I20260924 07:13:54.962739 22434 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd387d5ae0ca419f8ec7f545d58b7615 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd387d5ae0ca419f8ec7f545d58b7615" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd387d5ae0ca419f8ec7f545d58b7615" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43445 } } }
I20260924 07:13:54.962751 22435 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd387d5ae0ca419f8ec7f545d58b7615 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd387d5ae0ca419f8ec7f545d58b7615. Latest consensus state: current_term: 1 leader_uuid: "cd387d5ae0ca419f8ec7f545d58b7615" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd387d5ae0ca419f8ec7f545d58b7615" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43445 } } }
I20260924 07:13:54.962826 22434 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd387d5ae0ca419f8ec7f545d58b7615 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:54.962846 22435 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd387d5ae0ca419f8ec7f545d58b7615 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:54.963012 22442 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:13:54.963469 22442 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:13:54.964120 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 22362
I20260924 07:13:54.964262 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:13:54.964468 392 external_mini_cluster.cc:1428] Setting key 463093ed36df1b068fcaf0579dbabede
I20260924 07:13:54.965060 22442 catalog_manager.cc:1347] Generated new cluster ID: 8d90ce9eeba444a0a63fecff2236addf
I20260924 07:13:54.965107 22442 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:13:54.974973 22453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:54.965073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46291 (local address 127.0.98.62:43445)
0924 07:13:54.965241 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:54.965249 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:54.969067 (+ 3818us) server_negotiation.cc:407] Connection header received
0924 07:13:54.969187 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:54.969194 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:54.969249 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:54.969389 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:54.969640 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:54.971452 (+ 1812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:54.972512 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:54.973164 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:54.973274 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:54.974156 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:54.974160 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:54.974161 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:13:54.974373 (+ 212us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:54.974411 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:13:54.974425 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:54.974509 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:54.974572 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 07:13:54.974750 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":69,"threads_started":1}
I20260924 07:13:54.975803 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:13:54.980492 22453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:54.977341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57497 (local address 127.0.98.62:43445)
0924 07:13:54.977373 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:54.977374 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:54.977382 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:13:54.977424 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:54.977424 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:54.977434 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:54.977504 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:54.977598 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:54.978309 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:54.979274 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:54.979577 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:54.979612 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:54.980253 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:54.980255 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:54.980255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:13:54.980259 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:54.980273 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:13:54.980277 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:54.980336 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:54.980393 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 07:13:54.980415 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 07:13:54.984884 22442 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:13:54.985383 22442 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:13:54.998610 22442 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd387d5ae0ca419f8ec7f545d58b7615: Generated new TSK 0
I20260924 07:13:54.998836 22442 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:13:55.003955 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 22362
2026-09-24T07:13:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1368 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:13:55 dist-test-slave-14ww krb5kdc[22461](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:13:55 dist-test-slave-14ww krb5kdc[22461](info): set up 2 sockets
Sep 24 07:13:55 dist-test-slave-14ww krb5kdc[22461](info): commencing operation
krb5kdc: starting...
W20260924 07:13:56.068976 392 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:13:56 dist-test-slave-14ww krb5kdc[22461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:13:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:13:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:13:56.209565 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41685
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:56.290221 22474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:56.290428 22474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:56.290450 22474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:56.291617 22474 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:13:56.291659 22474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:56.291674 22474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:56.291687 22474 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:13:56.291699 22474 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:13:56.293231 22474 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:56.293478 22474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:56.293754 22474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:56.298633 22479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:56.298626 22480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:56.298789 22474 server_base.cc:1034] running on GCE node
W20260924 07:13:56.298672 22482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:56.299119 22474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:56.299350 22474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:56.300498 22474 hybrid_clock.cc:648] HybridClock initialized: now 1790234036300482 us; error 35 us; skew 500 ppm
Sep 24 07:13:56 dist-test-slave-14ww krb5kdc[22461](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:56.303018 22474 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:13:56.303458 22474 webserver.cc:460] Webserver started at http://127.0.98.62:37929/ using document root <none> and password file <none>
I20260924 07:13:56.303630 22474 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:56.303669 22474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:56.303783 22474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:56.304867 22474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "7aabe4d5a3a946e1a563fe24dfade255"
format_stamp: "Formatted at 2026-09-24 07:13:56 on dist-test-slave-14ww"
server_key: "bc2a70510796af5275b86449f1954603"
server_key_iv: "6ed0743b207b62eb70372a021c843013"
server_key_version: "encryptionkey@0"
I20260924 07:13:56.305177 22474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "7aabe4d5a3a946e1a563fe24dfade255"
format_stamp: "Formatted at 2026-09-24 07:13:56 on dist-test-slave-14ww"
server_key: "bc2a70510796af5275b86449f1954603"
server_key_iv: "6ed0743b207b62eb70372a021c843013"
server_key_version: "encryptionkey@0"
I20260924 07:13:56.306396 22474 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:13:56.307075 22489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:56.307245 22474 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 07:13:56.307349 22474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "7aabe4d5a3a946e1a563fe24dfade255"
format_stamp: "Formatted at 2026-09-24 07:13:56 on dist-test-slave-14ww"
server_key: "bc2a70510796af5275b86449f1954603"
server_key_iv: "6ed0743b207b62eb70372a021c843013"
server_key_version: "encryptionkey@0"
I20260924 07:13:56.307428 22474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:56.310756 22474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:56.313241 22474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:56.313405 22474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:56.317466 22474 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41685
I20260924 07:13:56.317579 22541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41685 every 8 connection(s)
I20260924 07:13:56.317873 22474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:13:56.318809 22542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:56.321161 22542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aabe4d5a3a946e1a563fe24dfade255: Bootstrap starting.
I20260924 07:13:56.321720 22542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7aabe4d5a3a946e1a563fe24dfade255: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:56.322016 22542 log.cc:824] T 00000000000000000000000000000000 P 7aabe4d5a3a946e1a563fe24dfade255: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:56.322687 22542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aabe4d5a3a946e1a563fe24dfade255: No bootstrap required, opened a new log
I20260924 07:13:56.324201 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 22474
I20260924 07:13:56.324361 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:13:56.324239 22542 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7aabe4d5a3a946e1a563fe24dfade255 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aabe4d5a3a946e1a563fe24dfade255" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41685 } }
I20260924 07:13:56.324409 22542 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7aabe4d5a3a946e1a563fe24dfade255 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 07:13:56.324476 22542 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7aabe4d5a3a946e1a563fe24dfade255 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aabe4d5a3a946e1a563fe24dfade255, State: Initialized, Role: FOLLOWER
I20260924 07:13:56.324589 392 external_mini_cluster.cc:1428] Setting key 96005a7b2dbc85785f924e63dbbf6c29
I20260924 07:13:56.324664 22542 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7aabe4d5a3a946e1a563fe24dfade255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aabe4d5a3a946e1a563fe24dfade255" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41685 } }
I20260924 07:13:56.324780 22542 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7aabe4d5a3a946e1a563fe24dfade255 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 07:13:56.324844 22542 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7aabe4d5a3a946e1a563fe24dfade255 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 07:13:56.324903 22542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7aabe4d5a3a946e1a563fe24dfade255 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:56.326079 22542 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7aabe4d5a3a946e1a563fe24dfade255 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aabe4d5a3a946e1a563fe24dfade255" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41685 } }
I20260924 07:13:56.326211 22542 leader_election.cc:302] T 00000000000000000000000000000000 P 7aabe4d5a3a946e1a563fe24dfade255 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7aabe4d5a3a946e1a563fe24dfade255; no voters:
I20260924 07:13:56.326359 22542 leader_election.cc:288] T 00000000000000000000000000000000 P 7aabe4d5a3a946e1a563fe24dfade255 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 07:13:56.326392 22547 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7aabe4d5a3a946e1a563fe24dfade255 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:13:56.326560 22547 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7aabe4d5a3a946e1a563fe24dfade255 [term 1 LEADER]: Becoming Leader. State: Replica: 7aabe4d5a3a946e1a563fe24dfade255, State: Running, Role: LEADER
I20260924 07:13:56.326678 22547 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7aabe4d5a3a946e1a563fe24dfade255 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aabe4d5a3a946e1a563fe24dfade255" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41685 } }
I20260924 07:13:56.326681 22542 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7aabe4d5a3a946e1a563fe24dfade255 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:13:56.327049 22549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aabe4d5a3a946e1a563fe24dfade255 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7aabe4d5a3a946e1a563fe24dfade255. Latest consensus state: current_term: 1 leader_uuid: "7aabe4d5a3a946e1a563fe24dfade255" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aabe4d5a3a946e1a563fe24dfade255" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41685 } } }
I20260924 07:13:56.327052 22548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aabe4d5a3a946e1a563fe24dfade255 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7aabe4d5a3a946e1a563fe24dfade255" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aabe4d5a3a946e1a563fe24dfade255" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41685 } } }
I20260924 07:13:56.327143 22548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aabe4d5a3a946e1a563fe24dfade255 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:56.327135 22549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aabe4d5a3a946e1a563fe24dfade255 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:56.327078 22546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:56.325146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34795 (local address 127.0.98.62:41685)
0924 07:13:56.325370 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:56.325383 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:56.325422 (+ 39us) server_negotiation.cc:407] Connection header received
0924 07:13:56.325560 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:56.325570 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:56.325619 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:56.325771 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:56.326443 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:56.326447 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:56.326448 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:13:56.326453 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:56.326502 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:13:56.326519 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:56.326804 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:56.326837 (+ 33us) server_negotiation.cc:299] Negotiation successful
0924 07:13:56.326880 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":114,"threads_started":1}
I20260924 07:13:56.328117 22551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:13:56.328228 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:13:56.328631 22551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:13:56.330078 22551 catalog_manager.cc:1347] Generated new cluster ID: 34ac33f63e4846ab8f3accb9d9332413
I20260924 07:13:56.330142 22551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:13:56.331245 22546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:56.329766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36477 (local address 127.0.98.62:41685)
0924 07:13:56.329799 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:56.329801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:56.330207 (+ 406us) server_negotiation.cc:407] Connection header received
0924 07:13:56.330317 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:56.330318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:56.330323 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:56.330351 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:56.331003 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:56.331004 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:56.331005 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 07:13:56.331009 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:56.331022 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 07:13:56.331026 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:56.331056 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:56.331162 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 07:13:56.331174 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 07:13:56.338888 22551 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:13:56.339043 22551 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:13:56.348769 22551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7aabe4d5a3a946e1a563fe24dfade255: Generated new TSK 0
I20260924 07:13:56.349016 22551 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:13:56.356331 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 22474
2026-09-24T07:13:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1352 ms)
[----------] 18 tests from AuthTokenIssuingTest (23781 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:13:56 dist-test-slave-14ww krb5kdc[22573](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:13:56 dist-test-slave-14ww krb5kdc[22573](info): set up 2 sockets
Sep 24 07:13:56 dist-test-slave-14ww krb5kdc[22573](info): commencing operation
krb5kdc: starting...
W20260924 07:13:57.396951 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:13:57 dist-test-slave-14ww krb5kdc[22573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:13:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:13:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:13:57.509415 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37975
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37975,127.0.98.61:40617,127.0.98.60:39771
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:13:57.587471 22586 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 07:13:57.587651 22586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:57.587667 22586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:57.587687 22586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:57.588699 22586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:13:57.588778 22586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:57.588824 22586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:57.588838 22586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:13:57.588850 22586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:13:57.590224 22586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37975,127.0.98.61:40617,127.0.98.60:39771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:57.590400 22586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:57.590559 22586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:57.595186 22591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:57.595188 22592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:57.595342 22594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:57.595382 22586 server_base.cc:1034] running on GCE node
I20260924 07:13:57.595620 22586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:57.595849 22586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:57.597009 22586 hybrid_clock.cc:648] HybridClock initialized: now 1790234037596995 us; error 45 us; skew 500 ppm
Sep 24 07:13:57 dist-test-slave-14ww krb5kdc[22573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:57.610451 22586 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:13:57.610894 22586 webserver.cc:460] Webserver started at http://127.0.98.62:41949/ using document root <none> and password file <none>
I20260924 07:13:57.611122 22586 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:57.611191 22586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:57.611342 22586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:57.612372 22586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "1270df80dbf2496aaa92dbafebd9a843"
format_stamp: "Formatted at 2026-09-24 07:13:57 on dist-test-slave-14ww"
server_key: "532ffff73f904618b09873a866bc72a4"
server_key_iv: "1e431e6ce7b66257fadf576a6f1c7d00"
server_key_version: "encryptionkey@0"
I20260924 07:13:57.612720 22586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "1270df80dbf2496aaa92dbafebd9a843"
format_stamp: "Formatted at 2026-09-24 07:13:57 on dist-test-slave-14ww"
server_key: "532ffff73f904618b09873a866bc72a4"
server_key_iv: "1e431e6ce7b66257fadf576a6f1c7d00"
server_key_version: "encryptionkey@0"
I20260924 07:13:57.614013 22586 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:13:57.614708 22601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:57.614877 22586 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:13:57.614970 22586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "1270df80dbf2496aaa92dbafebd9a843"
format_stamp: "Formatted at 2026-09-24 07:13:57 on dist-test-slave-14ww"
server_key: "532ffff73f904618b09873a866bc72a4"
server_key_iv: "1e431e6ce7b66257fadf576a6f1c7d00"
server_key_version: "encryptionkey@0"
I20260924 07:13:57.615051 22586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:57.626040 22586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:57.630288 22586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:57.630450 22586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:57.634615 22586 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37975
I20260924 07:13:57.634649 22653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37975 every 8 connection(s)
I20260924 07:13:57.634987 22586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:13:57.635959 22654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:57.637513 22654 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37975 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:13:57.643523 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 22586
I20260924 07:13:57.643615 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-0/wal/instance
Sep 24 07:13:57 dist-test-slave-14ww krb5kdc[22573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:13:57.643826 392 external_mini_cluster.cc:1428] Setting key 7905d5dd15ba6c329ab259824c96588e
I20260924 07:13:57.648576 22656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.638091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37975 (local address 127.0.0.1:53780)
0924 07:13:57.638236 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:57.638268 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:57.638728 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:57.638978 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:57.638980 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:57.639206 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:57.639353 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:57.639356 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.640318 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.640319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:57.641094 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:57.641096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.641564 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.641565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:57.641615 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:57.642650 (+ 1035us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:57.642658 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:57.644099 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:57.647138 (+ 3039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:57.647144 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:57.647146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:57.647335 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:57.647582 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:57.647583 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:57.647583 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:57.647675 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:57.648014 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:57.648020 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:57.648078 (+ 58us) client_negotiation.cc:769] Sending connection context
0924 07:13:57.648189 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 07:13:57.648412 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":92,"thread_start_us":67,"threads_started":1}
I20260924 07:13:57.648583 22657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.638237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53780 (local address 127.0.98.62:37975)
0924 07:13:57.638361 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:57.638364 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:57.638391 (+ 27us) server_negotiation.cc:407] Connection header received
0924 07:13:57.638857 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:57.638858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:57.638875 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:57.638948 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:57.639430 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.640217 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.641180 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.641471 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.641556 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:57.644187 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:57.644194 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:57.644194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:57.644201 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:57.647035 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:57.647429 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:57.647432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:57.647433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:57.647514 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:57.647740 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:57.647741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:57.647741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:57.647943 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:57.648036 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:57.648212 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 07:13:57.648351 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":63,"threads_started":1}
I20260924 07:13:57.649349 22654 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40617 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:13:57.649700 22656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.649574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40617 (local address 127.0.0.1:54006)
0924 07:13:57.649619 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:57.649681 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 07:13:57.649853 22603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40617: connect: Connection refused (error 111)
I20260924 07:13:57.650224 22656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.650151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40617 (local address 127.0.0.1:54012)
0924 07:13:57.650177 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:57.650208 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260924 07:13:57.650342 22654 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40617: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40617: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 07:13:57.667346 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40617
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37975,127.0.98.61:40617,127.0.98.60:39771
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:13:57.699550 22654 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40617 } attempt: 1
I20260924 07:13:57.700053 22656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.699916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40617 (local address 127.0.0.1:54016)
0924 07:13:57.699983 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:57.700025 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 07:13:57.700440 22656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.700333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40617 (local address 127.0.0.1:54032)
0924 07:13:57.700386 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:57.700412 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 07:13:57.700531 22654 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40617: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40617: connect: Connection refused (error 111)
W20260924 07:13:57.745801 22661 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 07:13:57.745992 22661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:57.746009 22661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:57.746062 22661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:57.747200 22661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:13:57.747239 22661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:57.747254 22661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:57.747308 22661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:13:57.747330 22661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:13:57.748816 22661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37975,127.0.98.61:40617,127.0.98.60:39771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:57.749051 22661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:57.749271 22661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:13:57.753927 22669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:57.753925 22667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:57.754001 22666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:57.754127 22661 server_base.cc:1034] running on GCE node
I20260924 07:13:57.754335 22661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:57.754552 22661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:57.755686 22661 hybrid_clock.cc:648] HybridClock initialized: now 1790234037755668 us; error 34 us; skew 500 ppm
Sep 24 07:13:57 dist-test-slave-14ww krb5kdc[22573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:57.758496 22661 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 07:13:57.758955 22661 webserver.cc:460] Webserver started at http://127.0.98.61:42899/ using document root <none> and password file <none>
I20260924 07:13:57.759164 22661 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:57.759230 22661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:57.759377 22661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:57.760576 22661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-1/data/instance:
uuid: "7eea757770ed4b3583b791f49039a950"
format_stamp: "Formatted at 2026-09-24 07:13:57 on dist-test-slave-14ww"
server_key: "923d42566f6599c039d635e0c36641ca"
server_key_iv: "bfe7e62ded82b946446fe21bc8b4e26b"
server_key_version: "encryptionkey@0"
I20260924 07:13:57.760968 22661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-1/wal/instance:
uuid: "7eea757770ed4b3583b791f49039a950"
format_stamp: "Formatted at 2026-09-24 07:13:57 on dist-test-slave-14ww"
server_key: "923d42566f6599c039d635e0c36641ca"
server_key_iv: "bfe7e62ded82b946446fe21bc8b4e26b"
server_key_version: "encryptionkey@0"
I20260924 07:13:57.762256 22661 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 07:13:57.762967 22676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:57.763135 22661 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 07:13:57.763234 22661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-1/wal
uuid: "7eea757770ed4b3583b791f49039a950"
format_stamp: "Formatted at 2026-09-24 07:13:57 on dist-test-slave-14ww"
server_key: "923d42566f6599c039d635e0c36641ca"
server_key_iv: "bfe7e62ded82b946446fe21bc8b4e26b"
server_key_version: "encryptionkey@0"
I20260924 07:13:57.763314 22661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:57.768666 22654 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40617 } attempt: 2
I20260924 07:13:57.769050 22656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.768937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40617 (local address 127.0.0.1:54038)
0924 07:13:57.768990 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:57.769024 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 07:13:57.769344 22656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.769249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40617 (local address 127.0.0.1:54042)
0924 07:13:57.769294 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:57.769325 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 07:13:57.769429 22654 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40617: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40617: connect: Connection refused (error 111)
I20260924 07:13:57.773689 22661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:57.776576 22661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:57.776741 22661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:57.780647 22661 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40617
I20260924 07:13:57.780675 22728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40617 every 8 connection(s)
I20260924 07:13:57.781016 22661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-1/data/info.pb
I20260924 07:13:57.781939 22729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:57.783319 22729 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37975 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:13:57 dist-test-slave-14ww krb5kdc[22573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:13:57.791294 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 22661
I20260924 07:13:57.791407 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-1/wal/instance
I20260924 07:13:57.791599 392 external_mini_cluster.cc:1428] Setting key b817687c454fb3ea13fc1fcae94c6be0
I20260924 07:13:57.793574 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.783925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37975 (local address 127.0.0.1:53790)
0924 07:13:57.784078 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:57.784116 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:57.784552 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:57.784840 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:57.784844 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:57.785165 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:57.785348 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:57.785352 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.786047 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.786047 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:57.786872 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:57.786875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.787232 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.787233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:57.787340 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:57.788245 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:57.788253 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:57.789594 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:57.792060 (+ 2466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:57.792068 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:57.792070 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:57.792386 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:57.792575 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:57.792576 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:57.792577 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:57.792822 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:57.793128 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:57.793133 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:57.793243 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 07:13:57.793320 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 07:13:57.793431 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260924 07:13:57.793581 22657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.783942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53790 (local address 127.0.98.62:37975)
0924 07:13:57.783998 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:57.784000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:57.784207 (+ 207us) server_negotiation.cc:407] Connection header received
0924 07:13:57.784628 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:57.784629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:57.784638 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:57.784751 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:57.785428 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.785976 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.786947 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.787131 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.787181 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:57.789708 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:57.789709 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:57.789709 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:57.789715 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:57.791977 (+ 2262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:57.792464 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:57.792464 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:57.792465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:57.792513 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:57.792899 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:57.792900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:57.792901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:57.793051 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:57.793125 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:57.793378 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 07:13:57.793465 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 07:13:57.794113 22729 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40617 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:13:57 dist-test-slave-14ww krb5kdc[22573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260924 07:13:57.803020 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.794423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54058 (local address 127.0.98.61:40617)
0924 07:13:57.794590 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:57.794593 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:57.794608 (+ 15us) server_negotiation.cc:407] Connection header received
0924 07:13:57.794624 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:57.794625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:57.794635 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:57.794710 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:57.794964 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.795641 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.796321 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.796628 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.796716 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:57.799090 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:57.799098 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:57.799098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:57.799104 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:57.801878 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:57.802221 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:57.802224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:57.802225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:57.802285 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:57.802483 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:57.802483 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:57.802484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:57.802621 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:57.802710 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:57.802821 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 07:13:57.802900 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":93,"threads_started":1}
I20260924 07:13:57.803037 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.794356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40617 (local address 127.0.0.1:54058)
0924 07:13:57.794402 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:57.794412 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:57.794447 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:57.794710 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:57.794711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:57.794851 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:57.794890 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:57.794891 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.795734 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.795735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:57.796225 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:57.796226 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.796714 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.796715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:57.796802 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:57.797890 (+ 1088us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:57.797895 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:57.798989 (+ 1094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:57.801967 (+ 2978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:57.801967 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:57.801968 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:57.802129 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:57.802349 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:57.802349 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:57.802350 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:57.802403 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:57.802689 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:57.802689 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:57.802725 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 07:13:57.802796 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 07:13:57.802948 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":69}
I20260924 07:13:57.803784 22729 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39771 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:13:57.804116 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.804003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39771 (local address 127.0.0.1:51632)
0924 07:13:57.804048 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:57.804094 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 07:13:57.804244 22678 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39771: connect: Connection refused (error 111)
I20260924 07:13:57.804621 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.804527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39771 (local address 127.0.0.1:51640)
0924 07:13:57.804576 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:57.804600 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 07:13:57.804723 22729 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39771: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39771: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 07:13:57.824469 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39771
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37975,127.0.98.61:40617,127.0.98.60:39771
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:13:57.853962 22729 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39771 } attempt: 1
I20260924 07:13:57.854509 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.854357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39771 (local address 127.0.0.1:51650)
0924 07:13:57.854426 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:57.854473 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 07:13:57.854887 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.854786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39771 (local address 127.0.0.1:51654)
0924 07:13:57.854834 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:57.854862 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 07:13:57.855018 22729 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39771: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39771: connect: Connection refused (error 111)
I20260924 07:13:57.860579 22654 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40617 } attempt: 3
Sep 24 07:13:57 dist-test-slave-14ww krb5kdc[22573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 07:13:57.869551 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.861016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54074 (local address 127.0.98.61:40617)
0924 07:13:57.861074 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:57.861076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:57.861087 (+ 11us) server_negotiation.cc:407] Connection header received
0924 07:13:57.861168 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:57.861169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:57.861185 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:57.861250 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:57.861577 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.862077 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.862799 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.863006 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.863094 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:57.865762 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:57.865763 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:57.865763 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:57.865769 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:57.868502 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:57.868901 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:57.868902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:57.868902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:57.868949 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:57.869146 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:57.869146 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:57.869146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:57.869233 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:57.869277 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:57.869436 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 07:13:57.869496 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 07:13:57.869604 22656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.860967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40617 (local address 127.0.0.1:54074)
0924 07:13:57.861025 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:57.861052 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:57.861102 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:57.861256 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:57.861257 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:57.861430 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:57.861494 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:57.861495 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.862170 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.862172 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:57.862708 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:57.862710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.863095 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.863096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:57.863179 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:57.864251 (+ 1072us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:57.864256 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:57.865684 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:57.868595 (+ 2911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:57.868596 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:57.868596 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:57.868830 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:57.869011 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:57.869012 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:57.869012 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:57.869078 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:57.869291 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:57.869292 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:57.869339 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 07:13:57.869421 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 07:13:57.869513 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 07:13:57.870110 22654 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39771 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:13:57.870463 22656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.870357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39771 (local address 127.0.0.1:51662)
0924 07:13:57.870403 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:57.870436 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 07:13:57.870741 22656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.870661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39771 (local address 127.0.0.1:51666)
0924 07:13:57.870700 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:57.870723 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 07:13:57.870836 22654 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39771: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39771: connect: Connection refused (error 111)
I20260924 07:13:57.902004 22654 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39771 } attempt: 1
I20260924 07:13:57.902506 22656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.902377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39771 (local address 127.0.0.1:51672)
0924 07:13:57.902441 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:57.902479 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 07:13:57.902882 22656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.902796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39771 (local address 127.0.0.1:51682)
0924 07:13:57.902840 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:57.902866 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 07:13:57.902992 22654 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39771: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39771: connect: Connection refused (error 111)
W20260924 07:13:57.907248 22736 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 07:13:57.907434 22736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:13:57.907451 22736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:13:57.907471 22736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:13:57.908535 22736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:13:57.908569 22736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:13:57.908583 22736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:13:57.908595 22736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:13:57.908607 22736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:13:57.910157 22736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37975,127.0.98.61:40617,127.0.98.60:39771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:13:57.910342 22736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:13:57.910499 22736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:13:57.915365 22736 server_base.cc:1034] running on GCE node
W20260924 07:13:57.915314 22742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:57.915323 22744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:13:57.915323 22741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:13:57.915771 22736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:13:57.916046 22736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:13:57.917212 22736 hybrid_clock.cc:648] HybridClock initialized: now 1790234037917207 us; error 32 us; skew 500 ppm
Sep 24 07:13:57 dist-test-slave-14ww krb5kdc[22573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:13:57.920401 22736 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 07:13:57.920882 22736 webserver.cc:460] Webserver started at http://127.0.98.60:38107/ using document root <none> and password file <none>
I20260924 07:13:57.921104 22736 fs_manager.cc:359] Metadata directory not provided
I20260924 07:13:57.921170 22736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:13:57.921322 22736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:13:57.922323 22736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-2/data/instance:
uuid: "20446120f1474152aeac1de851abcbd5"
format_stamp: "Formatted at 2026-09-24 07:13:57 on dist-test-slave-14ww"
server_key: "2289b2c27c71007294ff5191d1f30b97"
server_key_iv: "cde9cf814feeb96771eee16c9b0a3370"
server_key_version: "encryptionkey@0"
I20260924 07:13:57.922685 22736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-2/wal/instance:
uuid: "20446120f1474152aeac1de851abcbd5"
format_stamp: "Formatted at 2026-09-24 07:13:57 on dist-test-slave-14ww"
server_key: "2289b2c27c71007294ff5191d1f30b97"
server_key_iv: "cde9cf814feeb96771eee16c9b0a3370"
server_key_version: "encryptionkey@0"
I20260924 07:13:57.923171 22729 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39771 } attempt: 2
I20260924 07:13:57.923552 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.923411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39771 (local address 127.0.0.1:51690)
0924 07:13:57.923479 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:57.923517 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 07:13:57.923959 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.923862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39771 (local address 127.0.0.1:51700)
0924 07:13:57.923911 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:57.923936 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 07:13:57.924067 22729 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39771: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39771: connect: Connection refused (error 111)
I20260924 07:13:57.924304 22736 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 07:13:57.925107 22751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:13:57.925313 22736 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 07:13:57.925384 22736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-2/wal
uuid: "20446120f1474152aeac1de851abcbd5"
format_stamp: "Formatted at 2026-09-24 07:13:57 on dist-test-slave-14ww"
server_key: "2289b2c27c71007294ff5191d1f30b97"
server_key_iv: "cde9cf814feeb96771eee16c9b0a3370"
server_key_version: "encryptionkey@0"
I20260924 07:13:57.925470 22736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:13:57.943409 22736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:13:57.947597 22736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:13:57.947763 22736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:13:57.953007 22736 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39771
I20260924 07:13:57.953042 22803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39771 every 8 connection(s)
I20260924 07:13:57.953433 22736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-2/data/info.pb
I20260924 07:13:57.954349 22804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:13:57.955873 22804 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37975 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:13:57.959013 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 22736
I20260924 07:13:57.959105 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790233772407728-392-0/minicluster-data/master-2/wal/instance
I20260924 07:13:57.959275 392 external_mini_cluster.cc:1428] Setting key 08a398e8565b2a58bed57bbbfbd921bd
Sep 24 07:13:57 dist-test-slave-14ww krb5kdc[22573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 07:13:57 dist-test-slave-14ww krb5kdc[22573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 07:13:57.967484 22657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.956518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53804 (local address 127.0.98.62:37975)
0924 07:13:57.956585 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:57.956588 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:57.956839 (+ 251us) server_negotiation.cc:407] Connection header received
0924 07:13:57.957266 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:57.957268 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:57.957278 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:57.957329 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:57.957841 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.958306 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.959516 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.959810 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.959895 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:57.963025 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:57.963026 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:57.963026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:57.963034 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:57.966063 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:57.966522 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:57.966523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:57.966524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:57.966584 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:57.966871 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:57.966872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:57.966872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:57.966984 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:57.967053 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:57.967317 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 07:13:57.967396 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 07:13:57.967661 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.956501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37975 (local address 127.0.0.1:53804)
0924 07:13:57.956685 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:57.956717 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:57.957186 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:57.957376 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:57.957379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:57.957591 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:57.957774 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:57.957777 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.958379 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.958379 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:57.959429 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:57.959432 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.959920 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.959921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:57.960081 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:57.961280 (+ 1199us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:57.961289 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:57.962941 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:57.966131 (+ 3190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:57.966138 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:57.966140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:57.966445 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:57.966652 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:57.966653 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:57.966653 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:57.966810 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:57.967057 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:57.967063 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:57.967191 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 07:13:57.967295 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 07:13:57.967430 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":100,"threads_started":1}
I20260924 07:13:57.968116 22804 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40617 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:13:57.970912 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.959625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51710 (local address 127.0.98.60:39771)
0924 07:13:57.959985 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:57.959989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:57.960006 (+ 17us) server_negotiation.cc:407] Connection header received
0924 07:13:57.960026 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:57.960028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:57.960051 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:57.960125 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:57.960973 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.962018 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.962919 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.963273 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.963354 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:57.966471 (+ 3117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:57.966479 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:57.966479 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:57.966486 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:57.969753 (+ 3267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:57.970084 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:57.970087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:57.970088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:57.970143 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:57.970349 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:57.970350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:57.970350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:57.970470 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:57.970560 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:57.970702 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 07:13:57.970786 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":88,"threads_started":1}
Sep 24 07:13:57 dist-test-slave-14ww krb5kdc[22573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 24 07:13:57 dist-test-slave-14ww krb5kdc[22573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:13:57.977219 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.968346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40617 (local address 127.0.0.1:54084)
0924 07:13:57.968459 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:57.968469 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:57.968517 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:57.968638 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:57.968638 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:57.968827 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:57.968878 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:57.968880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.969799 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.969799 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:57.970233 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:57.970234 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.970697 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.970699 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:57.970771 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:57.971958 (+ 1187us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:57.971963 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:57.973316 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:57.976201 (+ 2885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:57.976202 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:57.976202 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:57.976351 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:57.976561 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:57.976562 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:57.976562 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:57.976621 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:57.976931 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:57.976932 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:57.976971 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 07:13:57.977050 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 07:13:57.977137 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260924 07:13:57.977264 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.968414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54084 (local address 127.0.98.61:40617)
0924 07:13:57.968461 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:57.968463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:57.968506 (+ 43us) server_negotiation.cc:407] Connection header received
0924 07:13:57.968571 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:57.968572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:57.968583 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:57.968636 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:57.968957 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.969689 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.970288 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.970592 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.970667 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:57.973391 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:57.973392 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:57.973393 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:57.973399 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:57.976121 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:57.976430 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:57.976431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:57.976432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:57.976493 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:57.976685 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:57.976686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:57.976687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:57.976875 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:57.976938 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:57.977089 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 07:13:57.977160 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 07:13:57.977509 22804 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39771 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:13:57.978125 22654 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39771 } attempt: 2
I20260924 07:13:57.981451 22657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.971632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53812 (local address 127.0.98.62:37975)
0924 07:13:57.971669 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:57.971671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:57.971688 (+ 17us) server_negotiation.cc:407] Connection header received
0924 07:13:57.971708 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:57.971709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:57.971718 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:57.971764 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:57.972347 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.973045 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.973921 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.974228 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.974314 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:57.977205 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:57.977206 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:57.977207 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:57.977220 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:57.980316 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:57.980653 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:57.980654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:57.980654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:57.980709 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:57.980939 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:57.980940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:57.980941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:57.981069 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:57.981225 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:57.981272 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 07:13:57.981347 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Sep 24 07:13:57 dist-test-slave-14ww krb5kdc[22573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 07:13:57 dist-test-slave-14ww krb5kdc[22573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 07:13:57 dist-test-slave-14ww krb5kdc[22573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 07:13:57.989367 22806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.977725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39771 (local address 127.0.0.1:51720)
0924 07:13:57.977860 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:57.977874 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:57.977929 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:57.978050 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:57.978051 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:57.978207 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:57.978246 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:57.978248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.979131 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.979132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:57.979910 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:57.979913 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.980679 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.980680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:57.980767 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:57.982419 (+ 1652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:57.982423 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:57.984072 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:57.988025 (+ 3953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:57.988026 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:57.988026 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:57.988423 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:57.988614 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:57.988615 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:57.988615 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:57.988682 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:57.989032 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:57.989032 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:57.989075 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 07:13:57.989141 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 07:13:57.989245 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260924 07:13:57.989548 22656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.978348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39771 (local address 127.0.0.1:51732)
0924 07:13:57.978384 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:57.978393 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:57.978429 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:57.978633 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:57.978634 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:57.978891 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:57.978944 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:57.978946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.979841 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.979842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:57.980583 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:57.980586 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.981055 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.981056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:57.981113 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:57.982410 (+ 1297us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:57.982415 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:57.984157 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:57.988463 (+ 4306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:57.988464 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:57.988464 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:57.988696 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:57.988921 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:57.988922 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:57.988922 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:57.988978 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:57.989244 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:57.989245 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:57.989285 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 07:13:57.989331 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 07:13:57.989430 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260924 07:13:57.989598 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.977757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51720 (local address 127.0.98.60:39771)
0924 07:13:57.977789 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:57.977791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:57.977917 (+ 126us) server_negotiation.cc:407] Connection header received
0924 07:13:57.977984 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:57.977985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:57.977993 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:57.978047 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:57.978325 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.979043 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.980001 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.980312 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.980376 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:57.984152 (+ 3776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:57.984153 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:57.984154 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:57.984160 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:57.987942 (+ 3782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:57.988505 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:57.988506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:57.988506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:57.988556 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:57.988791 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:57.988791 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:57.988792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:57.988951 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:57.989020 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:57.989182 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 07:13:57.989310 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":14}
I20260924 07:13:57.990199 22809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.978382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51732 (local address 127.0.98.60:39771)
0924 07:13:57.978511 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:57.978513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:57.978520 (+ 7us) server_negotiation.cc:407] Connection header received
0924 07:13:57.978536 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:57.978536 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:57.978545 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:57.978729 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:57.979027 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.979739 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.980661 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.980997 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.981055 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:57.984257 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:57.984258 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:57.984259 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:57.984264 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:57.988380 (+ 4116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:57.988806 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:57.988807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:57.988807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:57.988863 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:57.989052 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:57.989053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:57.989053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:57.989166 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:57.989926 (+ 760us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:57.989982 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 07:13:57.990055 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
I20260924 07:13:57.991006 22804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20446120f1474152aeac1de851abcbd5: Bootstrap starting.
I20260924 07:13:57.991616 22804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20446120f1474152aeac1de851abcbd5: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:57.991930 22804 log.cc:824] T 00000000000000000000000000000000 P 20446120f1474152aeac1de851abcbd5: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:57.992146 22654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1270df80dbf2496aaa92dbafebd9a843: Bootstrap starting.
I20260924 07:13:57.992664 22804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20446120f1474152aeac1de851abcbd5: No bootstrap required, opened a new log
I20260924 07:13:57.992944 22654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1270df80dbf2496aaa92dbafebd9a843: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:57.993221 22654 log.cc:824] T 00000000000000000000000000000000 P 1270df80dbf2496aaa92dbafebd9a843: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:57.993994 22654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1270df80dbf2496aaa92dbafebd9a843: No bootstrap required, opened a new log
I20260924 07:13:57.994519 22804 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20446120f1474152aeac1de851abcbd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1270df80dbf2496aaa92dbafebd9a843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37975 } } peers { permanent_uuid: "7eea757770ed4b3583b791f49039a950" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40617 } } peers { permanent_uuid: "20446120f1474152aeac1de851abcbd5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39771 } }
I20260924 07:13:57.994658 22804 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20446120f1474152aeac1de851abcbd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20446120f1474152aeac1de851abcbd5, State: Initialized, Role: FOLLOWER
I20260924 07:13:57.994745 22804 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20446120f1474152aeac1de851abcbd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1270df80dbf2496aaa92dbafebd9a843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37975 } } peers { permanent_uuid: "7eea757770ed4b3583b791f49039a950" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40617 } } peers { permanent_uuid: "20446120f1474152aeac1de851abcbd5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39771 } }
I20260924 07:13:57.995043 22804 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20446120f1474152aeac1de851abcbd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:13:57.994997 22812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20446120f1474152aeac1de851abcbd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1270df80dbf2496aaa92dbafebd9a843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37975 } } peers { permanent_uuid: "7eea757770ed4b3583b791f49039a950" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40617 } } peers { permanent_uuid: "20446120f1474152aeac1de851abcbd5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39771 } } }
I20260924 07:13:57.995160 22812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20446120f1474152aeac1de851abcbd5 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:13:57.995621 22654 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1270df80dbf2496aaa92dbafebd9a843 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1270df80dbf2496aaa92dbafebd9a843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37975 } } peers { permanent_uuid: "7eea757770ed4b3583b791f49039a950" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40617 } } peers { permanent_uuid: "20446120f1474152aeac1de851abcbd5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39771 } }
I20260924 07:13:57.995721 22654 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1270df80dbf2496aaa92dbafebd9a843 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1270df80dbf2496aaa92dbafebd9a843, State: Initialized, Role: FOLLOWER
I20260924 07:13:57.995819 22654 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1270df80dbf2496aaa92dbafebd9a843 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1270df80dbf2496aaa92dbafebd9a843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37975 } } peers { permanent_uuid: "7eea757770ed4b3583b791f49039a950" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40617 } } peers { permanent_uuid: "20446120f1474152aeac1de851abcbd5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39771 } }
I20260924 07:13:57.996080 22818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1270df80dbf2496aaa92dbafebd9a843 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1270df80dbf2496aaa92dbafebd9a843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37975 } } peers { permanent_uuid: "7eea757770ed4b3583b791f49039a950" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40617 } } peers { permanent_uuid: "20446120f1474152aeac1de851abcbd5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39771 } } }
I20260924 07:13:57.996155 22654 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1270df80dbf2496aaa92dbafebd9a843 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:13:57.996237 22818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1270df80dbf2496aaa92dbafebd9a843 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 07:13:57.996822 22824 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 20446120f1474152aeac1de851abcbd5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:13:57.996922 22824 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:13:57.997397 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:57.982131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54092 (local address 127.0.98.61:40617)
0924 07:13:57.982170 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:57.982172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:57.982181 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:13:57.982203 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:57.982203 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:57.982212 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:57.982281 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:57.982855 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.983575 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.984357 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:57.984546 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:57.984616 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:57.988355 (+ 3739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:57.988356 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:57.988357 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:57.988364 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:57.991916 (+ 3552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:57.992294 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:57.992295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:57.992296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:57.992354 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:57.996580 (+ 4226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:57.996581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:57.996581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:57.996705 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:57.997250 (+ 545us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:57.997283 (+ 33us) server_negotiation.cc:299] Negotiation successful
0924 07:13:57.997339 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 07:13:57.998564 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260924 07:13:57.998574 22836 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1270df80dbf2496aaa92dbafebd9a843: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:13:57.998617 22836 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:13:57.999548 22628 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1270df80dbf2496aaa92dbafebd9a843"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53812
I20260924 07:13:57.999675 22628 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1270df80dbf2496aaa92dbafebd9a843 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 07:13:57.999730 22628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1270df80dbf2496aaa92dbafebd9a843 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:57.997159 22736 master_runner.cc:186] Error getting master registration for 127.0.98.62:37975: OK, instance_id { permanent_uuid: "1270df80dbf2496aaa92dbafebd9a843" instance_seqno: 1790234037633501 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 07:13:58.000016 22736 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 07:13:58.000661 22628 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1270df80dbf2496aaa92dbafebd9a843 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1270df80dbf2496aaa92dbafebd9a843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37975 } } peers { permanent_uuid: "7eea757770ed4b3583b791f49039a950" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40617 } } peers { permanent_uuid: "20446120f1474152aeac1de851abcbd5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39771 } }
I20260924 07:13:58.000962 22628 leader_election.cc:288] T 00000000000000000000000000000000 P 1270df80dbf2496aaa92dbafebd9a843 [CANDIDATE]: Term 1 election: Requested vote from peers 7eea757770ed4b3583b791f49039a950 (127.0.98.61:40617), 20446120f1474152aeac1de851abcbd5 (127.0.98.60:39771)
I20260924 07:13:58.001487 22772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1270df80dbf2496aaa92dbafebd9a843" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "20446120f1474152aeac1de851abcbd5"
I20260924 07:13:58.001585 22772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20446120f1474152aeac1de851abcbd5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:58.002398 22772 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 20446120f1474152aeac1de851abcbd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1270df80dbf2496aaa92dbafebd9a843 in term 1.
I20260924 07:13:58.002784 22603 leader_election.cc:302] T 00000000000000000000000000000000 P 1270df80dbf2496aaa92dbafebd9a843 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1270df80dbf2496aaa92dbafebd9a843, 20446120f1474152aeac1de851abcbd5; no voters:
I20260924 07:13:58.003721 22818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1270df80dbf2496aaa92dbafebd9a843 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:13:58.004061 22818 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1270df80dbf2496aaa92dbafebd9a843 [term 1 LEADER]: Becoming Leader. State: Replica: 1270df80dbf2496aaa92dbafebd9a843, State: Running, Role: LEADER
I20260924 07:13:58.001997 22586 master_runner.cc:186] Error getting master registration for 127.0.98.61:40617: OK, instance_id { permanent_uuid: "7eea757770ed4b3583b791f49039a950" instance_seqno: 1790234037779549 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 07:13:58.004237 22586 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 07:13:58.004280 22818 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1270df80dbf2496aaa92dbafebd9a843 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1270df80dbf2496aaa92dbafebd9a843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37975 } } peers { permanent_uuid: "7eea757770ed4b3583b791f49039a950" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40617 } } peers { permanent_uuid: "20446120f1474152aeac1de851abcbd5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39771 } }
I20260924 07:13:58.004865 22818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1270df80dbf2496aaa92dbafebd9a843 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1270df80dbf2496aaa92dbafebd9a843. Latest consensus state: current_term: 1 leader_uuid: "1270df80dbf2496aaa92dbafebd9a843" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1270df80dbf2496aaa92dbafebd9a843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37975 } } peers { permanent_uuid: "7eea757770ed4b3583b791f49039a950" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40617 } } peers { permanent_uuid: "20446120f1474152aeac1de851abcbd5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39771 } } }
I20260924 07:13:58.004961 22818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1270df80dbf2496aaa92dbafebd9a843 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:58.005360 22844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:13:58.005560 22844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:13:58.007194 22772 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 20446120f1474152aeac1de851abcbd5 [term 1 FOLLOWER]: Refusing update from remote peer 1270df80dbf2496aaa92dbafebd9a843: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:13:58.007858 22818 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1270df80dbf2496aaa92dbafebd9a843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20446120f1474152aeac1de851abcbd5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:13:58.009279 22812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20446120f1474152aeac1de851abcbd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1270df80dbf2496aaa92dbafebd9a843. Latest consensus state: current_term: 1 leader_uuid: "1270df80dbf2496aaa92dbafebd9a843" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1270df80dbf2496aaa92dbafebd9a843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37975 } } peers { permanent_uuid: "7eea757770ed4b3583b791f49039a950" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40617 } } peers { permanent_uuid: "20446120f1474152aeac1de851abcbd5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39771 } } }
I20260924 07:13:58.009364 22812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20446120f1474152aeac1de851abcbd5 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:13:58.015221 22729 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39771 } attempt: 3
I20260924 07:13:58.019240 22842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1270df80dbf2496aaa92dbafebd9a843 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1270df80dbf2496aaa92dbafebd9a843" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1270df80dbf2496aaa92dbafebd9a843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37975 } } peers { permanent_uuid: "7eea757770ed4b3583b791f49039a950" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40617 } } peers { permanent_uuid: "20446120f1474152aeac1de851abcbd5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39771 } } }
I20260924 07:13:58.019353 22842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1270df80dbf2496aaa92dbafebd9a843 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:58.019951 22812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20446120f1474152aeac1de851abcbd5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1270df80dbf2496aaa92dbafebd9a843" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1270df80dbf2496aaa92dbafebd9a843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37975 } } peers { permanent_uuid: "7eea757770ed4b3583b791f49039a950" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40617 } } peers { permanent_uuid: "20446120f1474152aeac1de851abcbd5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39771 } } }
I20260924 07:13:58.020033 22812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20446120f1474152aeac1de851abcbd5 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:13:58.020277 22845 mvcc.cc:205] Tried to move back new op lower bound from 7332798619675078656 to 7332798619666927616. Current Snapshot: MvccSnapshot[applied={T|T < 7332798619675078656}]
Sep 24 07:13:58 dist-test-slave-14ww krb5kdc[22573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 07:13:58.023161 22844 catalog_manager.cc:1347] Generated new cluster ID: 42c6a07bd00742f99a49aefe137cb151
I20260924 07:13:58.023211 22844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:13:58.031352 22733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:58.006489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54096 (local address 127.0.98.61:40617)
0924 07:13:58.006527 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:58.006529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:58.016891 (+ 10362us) server_negotiation.cc:407] Connection header received
0924 07:13:58.017007 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:58.017008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:58.017021 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:58.017053 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:58.018597 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:58.019334 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:58.022202 (+ 2868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:58.022615 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:58.022661 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:58.025824 (+ 3163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:58.025826 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:58.025826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:58.025833 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:58.029525 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:58.030066 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:58.030068 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:58.030068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:58.030123 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:58.030381 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:58.030382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:58.030383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:58.030514 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:58.030604 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:58.030896 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 07:13:58.031252 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 07:13:58.032262 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:58.015472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39771 (local address 127.0.0.1:51752)
0924 07:13:58.015551 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0924 07:13:58.015561 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 07:13:58.015597 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:13:58.015866 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:13:58.015867 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:13:58.016031 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:13:58.016074 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:58.016075 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:58.017013 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:58.017014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:58.017808 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:13:58.017810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:13:58.018426 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:13:58.018426 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:13:58.018493 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:58.019735 (+ 1242us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:13:58.019740 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:13:58.021271 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:13:58.030736 (+ 9465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:58.030737 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:58.030737 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:58.030964 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:58.031559 (+ 595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:13:58.031559 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:13:58.031560 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:13:58.031632 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:13:58.031966 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:13:58.031967 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:13:58.032021 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 07:13:58.032081 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 07:13:58.032170 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 07:13:58.032564 22808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:58.015663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51752 (local address 127.0.98.60:39771)
0924 07:13:58.015701 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:58.015702 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:58.015709 (+ 7us) server_negotiation.cc:407] Connection header received
0924 07:13:58.015729 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:58.015730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:58.015751 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:58.015796 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:58.016175 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:58.016918 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:58.017996 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:58.018304 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:58.018365 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:58.021353 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:58.021354 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:58.021354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:58.021361 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:58.030618 (+ 9257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:58.031052 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:58.031053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:58.031054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:58.031121 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:58.031727 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:58.031728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:58.031728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:58.031870 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:58.031928 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:58.032382 (+ 454us) server_negotiation.cc:299] Negotiation successful
0924 07:13:58.032476 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 07:13:58.034159 22729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eea757770ed4b3583b791f49039a950: Bootstrap starting.
I20260924 07:13:58.034886 22729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7eea757770ed4b3583b791f49039a950: Neither blocks nor log segments found. Creating new log.
I20260924 07:13:58.035156 22729 log.cc:824] T 00000000000000000000000000000000 P 7eea757770ed4b3583b791f49039a950: Log is configured to *not* fsync() on all Append() calls
I20260924 07:13:58.036140 22729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eea757770ed4b3583b791f49039a950: No bootstrap required, opened a new log
I20260924 07:13:58.038127 22729 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7eea757770ed4b3583b791f49039a950 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1270df80dbf2496aaa92dbafebd9a843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37975 } } peers { permanent_uuid: "7eea757770ed4b3583b791f49039a950" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40617 } } peers { permanent_uuid: "20446120f1474152aeac1de851abcbd5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39771 } }
I20260924 07:13:58.038251 22729 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7eea757770ed4b3583b791f49039a950 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eea757770ed4b3583b791f49039a950, State: Initialized, Role: FOLLOWER
I20260924 07:13:58.038321 22729 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7eea757770ed4b3583b791f49039a950 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1270df80dbf2496aaa92dbafebd9a843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37975 } } peers { permanent_uuid: "7eea757770ed4b3583b791f49039a950" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40617 } } peers { permanent_uuid: "20446120f1474152aeac1de851abcbd5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39771 } }
I20260924 07:13:58.038622 22729 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7eea757770ed4b3583b791f49039a950 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:13:58.038743 22702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7eea757770ed4b3583b791f49039a950 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:13:58.039588 22702 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7eea757770ed4b3583b791f49039a950 [term 1 FOLLOWER]: Refusing update from remote peer 1270df80dbf2496aaa92dbafebd9a843: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:13:58.039844 22703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1270df80dbf2496aaa92dbafebd9a843" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7eea757770ed4b3583b791f49039a950"
I20260924 07:13:58.040215 22818 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1270df80dbf2496aaa92dbafebd9a843 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eea757770ed4b3583b791f49039a950" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:13:58.041963 22703 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7eea757770ed4b3583b791f49039a950 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1270df80dbf2496aaa92dbafebd9a843 in term 1.
W20260924 07:13:58.043491 22866 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7eea757770ed4b3583b791f49039a950: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:13:58.043620 22866 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:13:58.043808 22855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eea757770ed4b3583b791f49039a950 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1270df80dbf2496aaa92dbafebd9a843" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1270df80dbf2496aaa92dbafebd9a843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37975 } } peers { permanent_uuid: "7eea757770ed4b3583b791f49039a950" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40617 } } peers { permanent_uuid: "20446120f1474152aeac1de851abcbd5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39771 } } }
I20260924 07:13:58.043932 22855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eea757770ed4b3583b791f49039a950 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:13:58.044018 22855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eea757770ed4b3583b791f49039a950 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1270df80dbf2496aaa92dbafebd9a843. Latest consensus state: current_term: 1 leader_uuid: "1270df80dbf2496aaa92dbafebd9a843" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1270df80dbf2496aaa92dbafebd9a843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37975 } } peers { permanent_uuid: "7eea757770ed4b3583b791f49039a950" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40617 } } peers { permanent_uuid: "20446120f1474152aeac1de851abcbd5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39771 } } }
I20260924 07:13:58.044126 22855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eea757770ed4b3583b791f49039a950 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:13:58.044202 22855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eea757770ed4b3583b791f49039a950 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1270df80dbf2496aaa92dbafebd9a843" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1270df80dbf2496aaa92dbafebd9a843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37975 } } peers { permanent_uuid: "7eea757770ed4b3583b791f49039a950" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40617 } } peers { permanent_uuid: "20446120f1474152aeac1de851abcbd5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39771 } } }
I20260924 07:13:58.044294 22855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eea757770ed4b3583b791f49039a950 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:13:58.045552 22868 mvcc.cc:205] Tried to move back new op lower bound from 7332798619675078656 to 7332798619666927616. Current Snapshot: MvccSnapshot[applied={T|T < 7332798619675078656}]
I20260924 07:13:58.046255 22657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:58.005318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53822 (local address 127.0.98.62:37975)
0924 07:13:58.005356 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:58.005358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:58.007989 (+ 2631us) server_negotiation.cc:407] Connection header received
0924 07:13:58.008089 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:58.008090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:58.008100 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:58.008131 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:58.008972 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:58.009719 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:58.010755 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:58.011081 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:58.011150 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:58.014533 (+ 3383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:58.014535 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:58.014535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:58.014542 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:58.018239 (+ 3697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:58.038011 (+ 19772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:58.038013 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:58.038014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:58.038077 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:58.045617 (+ 7540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:58.045618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:58.045618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:58.045778 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:58.045875 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:58.046074 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 07:13:58.046163 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 07:13:58.050810 22809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:58.008211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51736 (local address 127.0.98.60:39771)
0924 07:13:58.008323 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:58.008326 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:58.017842 (+ 9516us) server_negotiation.cc:407] Connection header received
0924 07:13:58.017938 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:58.017939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:58.017951 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:58.017981 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:58.020408 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:58.021156 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:58.022163 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:58.022491 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:58.022556 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:58.046554 (+ 23998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:13:58.046556 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:13:58.046556 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:13:58.046563 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:13:58.049608 (+ 3045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:58.050004 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:58.050005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:58.050005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:58.050055 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:13:58.050243 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:13:58.050244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:13:58.050244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:13:58.050362 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:13:58.050427 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:58.050627 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 07:13:58.050720 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260924 07:13:58.056957 22818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1270df80dbf2496aaa92dbafebd9a843 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1270df80dbf2496aaa92dbafebd9a843" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1270df80dbf2496aaa92dbafebd9a843" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37975 } } peers { permanent_uuid: "7eea757770ed4b3583b791f49039a950" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40617 } } peers { permanent_uuid: "20446120f1474152aeac1de851abcbd5" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39771 } } }
I20260924 07:13:58.057057 22818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1270df80dbf2496aaa92dbafebd9a843 [sys.catalog]: This master's current role is: LEADER
I20260924 07:13:58.057957 22844 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:13:58.058661 22844 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:13:58.065814 22844 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1270df80dbf2496aaa92dbafebd9a843: Generated new TSK 0
I20260924 07:13:58.065987 22844 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 07:13:58.103408 22657 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:13:58.101468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53826 (local address 127.0.98.62:37975)
0924 07:13:58.101571 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:58.101573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:58.102752 (+ 1179us) server_negotiation.cc:407] Connection header received
0924 07:13:58.102858 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:58.102859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:58.102870 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:58.102900 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:58.103347 (+ 447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53826: BlockingRecv error: recv got EOF from 127.0.0.1:53826 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260924 07:13:58.104100 22733 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:13:58.102230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54100 (local address 127.0.98.61:40617)
0924 07:13:58.102268 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:58.102270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:58.103629 (+ 1359us) server_negotiation.cc:407] Connection header received
0924 07:13:58.103725 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:58.103726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:58.103737 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:58.103765 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:58.104042 (+ 277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54100: BlockingRecv error: recv got EOF from 127.0.0.1:54100 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 07:13:58.104841 22809 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:13:58.102506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51758 (local address 127.0.98.60:39771)
0924 07:13:58.102544 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:58.102546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:58.104335 (+ 1789us) server_negotiation.cc:407] Connection header received
0924 07:13:58.104430 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:58.104430 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:58.104440 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:58.104468 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:58.104774 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51758: BlockingRecv error: recv got EOF from 127.0.0.1:51758 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 07:13:58.106993 22733 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:13:58.106304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54102 (local address 127.0.98.61:40617)
0924 07:13:58.106338 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:58.106339 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:58.106499 (+ 160us) server_negotiation.cc:407] Connection header received
0924 07:13:58.106595 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:58.106596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:58.106613 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:58.106641 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:58.106951 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54102: BlockingRecv error: recv got EOF from 127.0.0.1:54102 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 07:13:58.110003 22733 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:13:58.109231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54104 (local address 127.0.98.61:40617)
0924 07:13:58.109265 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:58.109267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:58.109489 (+ 222us) server_negotiation.cc:407] Connection header received
0924 07:13:58.109586 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:58.109587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:58.109597 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:58.109625 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:58.109964 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54104: BlockingRecv error: recv got EOF from 127.0.0.1:54104 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 07:13:58.113806 22733 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:13:58.113078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54114 (local address 127.0.98.61:40617)
0924 07:13:58.113109 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:58.113111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:58.113333 (+ 222us) server_negotiation.cc:407] Connection header received
0924 07:13:58.113424 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:58.113425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:58.113435 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:58.113462 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:58.113768 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54114: BlockingRecv error: recv got EOF from 127.0.0.1:54114 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 07:13:58.119681 22733 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:13:58.118902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54120 (local address 127.0.98.61:40617)
0924 07:13:58.118933 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:58.118934 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:58.119154 (+ 220us) server_negotiation.cc:407] Connection header received
0924 07:13:58.119265 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:58.119266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:58.119276 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:58.119329 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:58.119642 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54120: BlockingRecv error: recv got EOF from 127.0.0.1:54120 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 07:13:58.129688 22733 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:13:58.128858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54130 (local address 127.0.98.61:40617)
0924 07:13:58.128893 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:58.128895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:58.129169 (+ 274us) server_negotiation.cc:407] Connection header received
0924 07:13:58.129278 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:58.129278 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:58.129290 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:58.129319 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:58.129641 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54130: BlockingRecv error: recv got EOF from 127.0.0.1:54130 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 07:13:58.147908 22733 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:13:58.146991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54142 (local address 127.0.98.61:40617)
0924 07:13:58.147034 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:58.147036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:58.147316 (+ 280us) server_negotiation.cc:407] Connection header received
0924 07:13:58.147422 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:58.147423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:58.147435 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:58.147465 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:58.147851 (+ 386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54142: BlockingRecv error: recv got EOF from 127.0.0.1:54142 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 07:13:58.182210 22733 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:13:58.181258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54154 (local address 127.0.98.61:40617)
0924 07:13:58.181304 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:58.181306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:58.181461 (+ 155us) server_negotiation.cc:407] Connection header received
0924 07:13:58.181536 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:58.181538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:58.181549 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:58.181626 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:58.182139 (+ 513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54154: BlockingRecv error: recv got EOF from 127.0.0.1:54154 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260924 07:13:58.248224 22733 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:13:58.247406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54162 (local address 127.0.98.61:40617)
0924 07:13:58.247489 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:58.247492 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:58.247604 (+ 112us) server_negotiation.cc:407] Connection header received
0924 07:13:58.247708 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:58.247709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:58.247722 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:58.247799 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:58.248161 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54162: BlockingRecv error: recv got EOF from 127.0.0.1:54162 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260924 07:13:58.378026 22733 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:13:58.377363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54172 (local address 127.0.98.61:40617)
0924 07:13:58.377438 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:58.377440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:58.377528 (+ 88us) server_negotiation.cc:407] Connection header received
0924 07:13:58.377596 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:58.377597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:58.377609 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:58.377656 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:58.377956 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54172: BlockingRecv error: recv got EOF from 127.0.0.1:54172 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260924 07:13:58.636070 22733 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:13:58.635304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54178 (local address 127.0.98.61:40617)
0924 07:13:58.635394 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:58.635397 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:58.635433 (+ 36us) server_negotiation.cc:407] Connection header received
0924 07:13:58.635526 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:58.635527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:58.635541 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:58.635606 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:13:58.635970 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54178: BlockingRecv error: recv got EOF from 127.0.0.1:54178 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 07:13:58.997470 22824 catalog_manager.cc:1259] Loaded cluster ID: 42c6a07bd00742f99a49aefe137cb151
I20260924 07:13:58.997607 22824 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 20446120f1474152aeac1de851abcbd5: loading cluster ID for follower catalog manager: success
I20260924 07:13:58.998490 22824 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 20446120f1474152aeac1de851abcbd5: acquiring CA information for follower catalog manager: success
I20260924 07:13:58.998797 22824 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 20446120f1474152aeac1de851abcbd5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 07:13:59.044116 22866 catalog_manager.cc:1259] Loaded cluster ID: 42c6a07bd00742f99a49aefe137cb151
I20260924 07:13:59.044224 22866 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7eea757770ed4b3583b791f49039a950: loading cluster ID for follower catalog manager: success
I20260924 07:13:59.045089 22866 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7eea757770ed4b3583b791f49039a950: acquiring CA information for follower catalog manager: success
I20260924 07:13:59.045409 22866 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7eea757770ed4b3583b791f49039a950: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 07:13:59.151790 22937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:59.149259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54194 (local address 127.0.98.61:40617)
0924 07:13:59.149364 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:59.149366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:59.149423 (+ 57us) server_negotiation.cc:407] Connection header received
0924 07:13:59.149494 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:59.149494 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:59.149544 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:59.149625 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:13:59.149813 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:59.150318 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:59.151088 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:59.151274 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:59.151324 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:59.151465 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:13:59.151571 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:13:59.151654 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:59.151672 (+ 18us) server_negotiation.cc:299] Negotiation successful
0924 07:13:59.151693 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":59,"threads_started":1}
I20260924 07:13:59.155699 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:59.153655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51768 (local address 127.0.98.60:39771)
0924 07:13:59.153757 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:59.153759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:59.153766 (+ 7us) server_negotiation.cc:407] Connection header received
0924 07:13:59.153800 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:59.153801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:59.153832 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:59.153894 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:13:59.153959 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:59.154405 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:59.155074 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:59.155265 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:59.155313 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:59.155427 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:13:59.155533 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:13:59.155583 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:59.155624 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 07:13:59.155640 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":54,"threads_started":1}
I20260924 07:13:59.160580 22949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:59.157372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53842 (local address 127.0.98.62:37975)
0924 07:13:59.157478 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:59.157480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:59.157490 (+ 10us) server_negotiation.cc:407] Connection header received
0924 07:13:59.157536 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:59.157537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:59.157581 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:59.157671 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:13:59.157821 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:59.158573 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:59.159558 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:59.159840 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:59.159902 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:59.160146 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:13:59.160241 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:13:59.160289 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:59.160460 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 07:13:59.160484 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":65,"threads_started":1}
I20260924 07:13:59.160923 22937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:59.157514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54206 (local address 127.0.98.61:40617)
0924 07:13:59.157560 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:59.157562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:59.157947 (+ 385us) server_negotiation.cc:407] Connection header received
0924 07:13:59.158042 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:59.158043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:59.158054 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:59.158124 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:13:59.158285 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:59.159035 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:59.160111 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:59.160398 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:59.160454 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:59.160645 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:13:59.160675 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:13:59.160700 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:59.160829 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 07:13:59.160852 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 07:13:59.161329 22943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:13:59.157776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51780 (local address 127.0.98.60:39771)
0924 07:13:59.157814 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 07:13:59.157816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:13:59.158500 (+ 684us) server_negotiation.cc:407] Connection header received
0924 07:13:59.158595 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:13:59.158596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:13:59.158604 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:13:59.158630 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:13:59.158799 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:59.159474 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:59.160483 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:13:59.160801 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:13:59.160837 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:13:59.161064 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:13:59.161094 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:13:59.161121 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0924 07:13:59.161241 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 07:13:59.161261 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 07:13:59.177454 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 22586
I20260924 07:13:59.182909 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 22661
I20260924 07:13:59.187739 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 22736
2026-09-24T07:13:59Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2832 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 07:13:59 dist-test-slave-14ww krb5kdc[22954](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 07:13:59 dist-test-slave-14ww krb5kdc[22954](info): set up 2 sockets
Sep 24 07:13:59 dist-test-slave-14ww krb5kdc[22954](info): commencing operation
krb5kdc: starting...
W20260924 07:14:00.229143 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/krb5kdc/test-user.keytab.
Sep 24 07:14:00 dist-test-slave-14ww krb5kdc[22954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T07:14:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T07:14:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 07:14:00.347256 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40511
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40511,127.0.98.61:42243,127.0.98.60:37023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 07:14:00.430817 22967 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 07:14:00.431013 22967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:14:00.431030 22967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:14:00.431051 22967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:14:00.432168 22967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:14:00.432207 22967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:14:00.432222 22967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:14:00.432235 22967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:14:00.432281 22967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:14:00.433799 22967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40511,127.0.98.61:42243,127.0.98.60:37023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:14:00.434029 22967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:14:00.434279 22967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:14:00.439149 22975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:14:00.439199 22972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:14:00.439404 22967 server_base.cc:1034] running on GCE node
W20260924 07:14:00.439404 22973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:14:00.439702 22967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:14:00.439978 22967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:14:00.441138 22967 hybrid_clock.cc:648] HybridClock initialized: now 1790234040441134 us; error 37 us; skew 500 ppm
Sep 24 07:14:00 dist-test-slave-14ww krb5kdc[22954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234040, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:14:00.443486 22967 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 07:14:00.443948 22967 webserver.cc:460] Webserver started at http://127.0.98.62:35645/ using document root <none> and password file <none>
I20260924 07:14:00.444172 22967 fs_manager.cc:359] Metadata directory not provided
I20260924 07:14:00.444242 22967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:14:00.444396 22967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:14:00.445470 22967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-0/data/instance:
uuid: "b5109dea49c44314a9d2ca79d9f1432b"
format_stamp: "Formatted at 2026-09-24 07:14:00 on dist-test-slave-14ww"
server_key: "829382caa86b5e3f7f1706e911ca2472"
server_key_iv: "23715e11cfc39f9f2f117bb54ead8dad"
server_key_version: "encryptionkey@0"
I20260924 07:14:00.445812 22967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-0/wal/instance:
uuid: "b5109dea49c44314a9d2ca79d9f1432b"
format_stamp: "Formatted at 2026-09-24 07:14:00 on dist-test-slave-14ww"
server_key: "829382caa86b5e3f7f1706e911ca2472"
server_key_iv: "23715e11cfc39f9f2f117bb54ead8dad"
server_key_version: "encryptionkey@0"
I20260924 07:14:00.447088 22967 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:14:00.447880 22982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:14:00.448088 22967 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:14:00.448191 22967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-0/wal
uuid: "b5109dea49c44314a9d2ca79d9f1432b"
format_stamp: "Formatted at 2026-09-24 07:14:00 on dist-test-slave-14ww"
server_key: "829382caa86b5e3f7f1706e911ca2472"
server_key_iv: "23715e11cfc39f9f2f117bb54ead8dad"
server_key_version: "encryptionkey@0"
I20260924 07:14:00.448263 22967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:14:00.461318 22967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:14:00.462980 22967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:14:00.463128 22967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:14:00.466975 22967 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40511
I20260924 07:14:00.467006 23034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40511 every 8 connection(s)
I20260924 07:14:00.467320 22967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-0/data/info.pb
I20260924 07:14:00.468210 23035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:14:00.469756 23035 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40511 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:14:00.471721 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 22967
I20260924 07:14:00.471846 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-0/wal/instance
I20260924 07:14:00.472045 392 external_mini_cluster.cc:1428] Setting key a8b9a8e082417415553d2cc33be00e58
Sep 24 07:14:00 dist-test-slave-14ww krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234040, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260924 07:14:00.480191 23037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.470306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40511 (local address 127.0.0.1:55018)
0924 07:14:00.470509 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:00.470552 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 07:14:00.471020 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:14:00.471309 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:14:00.471312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:14:00.471608 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:14:00.471837 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:00.471844 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.473008 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.473009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:00.473849 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:00.473851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.474572 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.474573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:00.474628 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:00.475133 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:14:00.475141 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:14:00.476542 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:14:00.478658 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:14:00.478664 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:14:00.478665 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:14:00.478889 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:14:00.479165 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:14:00.479166 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:14:00.479166 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:14:00.479247 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:14:00.479702 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:14:00.479710 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:14:00.479787 (+ 77us) client_negotiation.cc:769] Sending connection context
0924 07:14:00.479874 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 07:14:00.480000 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":80,"threads_started":1}
I20260924 07:14:00.480194 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.470376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55018 (local address 127.0.98.62:40511)
0924 07:14:00.470528 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:00.470538 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:00.470651 (+ 113us) server_negotiation.cc:407] Connection header received
0924 07:14:00.471178 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:00.471180 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:00.471196 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:00.471293 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:00.471962 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.472916 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.473939 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.474469 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.474577 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:00.476633 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:14:00.476640 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:14:00.476640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:14:00.476647 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:14:00.478557 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:00.478981 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:00.478985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:00.478985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:00.479085 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:00.479320 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:00.479321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:00.479321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:00.479596 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:14:00.479691 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:00.479931 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 07:14:00.480058 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
I20260924 07:14:00.481050 23035 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42243 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:14:00.481395 23037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.481286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42243 (local address 127.0.0.1:57512)
0924 07:14:00.481332 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:00.481375 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 07:14:00.481580 22984 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42243: connect: Connection refused (error 111)
I20260924 07:14:00.482070 23037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.481962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42243 (local address 127.0.0.1:57514)
0924 07:14:00.482014 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:00.482046 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 07:14:00.482223 23035 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42243: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42243: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 07:14:00.497815 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42243
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40511,127.0.98.61:42243,127.0.98.60:37023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:14:00.531431 23035 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42243 } attempt: 1
I20260924 07:14:00.531970 23037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.531791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42243 (local address 127.0.0.1:57516)
0924 07:14:00.531873 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:00.531928 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 07:14:00.532366 23037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.532246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42243 (local address 127.0.0.1:57522)
0924 07:14:00.532300 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:00.532334 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 07:14:00.532510 23035 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42243: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42243: connect: Connection refused (error 111)
W20260924 07:14:00.577347 23042 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 07:14:00.577562 23042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:14:00.577581 23042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:14:00.577600 23042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:14:00.578758 23042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:14:00.578797 23042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:14:00.578811 23042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:14:00.578823 23042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:14:00.578835 23042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:14:00.580323 23042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40511,127.0.98.61:42243,127.0.98.60:37023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:14:00.580569 23042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:14:00.580808 23042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 07:14:00.585528 23042 server_base.cc:1034] running on GCE node
W20260924 07:14:00.585482 23048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:14:00.585496 23047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:14:00.585497 23050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:14:00.585933 23042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:14:00.586208 23042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:14:00.587378 23042 hybrid_clock.cc:648] HybridClock initialized: now 1790234040587366 us; error 41 us; skew 500 ppm
Sep 24 07:14:00 dist-test-slave-14ww krb5kdc[22954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234040, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:14:00.589795 23042 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 07:14:00.590219 23042 webserver.cc:460] Webserver started at http://127.0.98.61:42867/ using document root <none> and password file <none>
I20260924 07:14:00.590443 23042 fs_manager.cc:359] Metadata directory not provided
I20260924 07:14:00.590512 23042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:14:00.590659 23042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:14:00.591738 23042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-1/data/instance:
uuid: "5559dfcea0fe4c4992d0aac393e77f02"
format_stamp: "Formatted at 2026-09-24 07:14:00 on dist-test-slave-14ww"
server_key: "0f2ddc7e571a0401eb6bf4cfaec78792"
server_key_iv: "c78ec94aa824a609c9c4678027f45d54"
server_key_version: "encryptionkey@0"
I20260924 07:14:00.592103 23042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-1/wal/instance:
uuid: "5559dfcea0fe4c4992d0aac393e77f02"
format_stamp: "Formatted at 2026-09-24 07:14:00 on dist-test-slave-14ww"
server_key: "0f2ddc7e571a0401eb6bf4cfaec78792"
server_key_iv: "c78ec94aa824a609c9c4678027f45d54"
server_key_version: "encryptionkey@0"
I20260924 07:14:00.593478 23042 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 07:14:00.594218 23057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:14:00.594405 23042 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 07:14:00.594503 23042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-1/wal
uuid: "5559dfcea0fe4c4992d0aac393e77f02"
format_stamp: "Formatted at 2026-09-24 07:14:00 on dist-test-slave-14ww"
server_key: "0f2ddc7e571a0401eb6bf4cfaec78792"
server_key_iv: "c78ec94aa824a609c9c4678027f45d54"
server_key_version: "encryptionkey@0"
I20260924 07:14:00.594586 23042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:14:00.600689 23035 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42243 } attempt: 2
I20260924 07:14:00.601140 23037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.600999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42243 (local address 127.0.0.1:57536)
0924 07:14:00.601057 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:00.601097 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 07:14:00.601490 23037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.601396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42243 (local address 127.0.0.1:57546)
0924 07:14:00.601441 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:00.601468 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 07:14:00.601614 23035 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42243: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42243: connect: Connection refused (error 111)
I20260924 07:14:00.610807 23042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:14:00.612530 23042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:14:00.612684 23042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:14:00.616449 23042 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42243
I20260924 07:14:00.616509 23109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42243 every 8 connection(s)
I20260924 07:14:00.616899 23042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-1/data/info.pb
I20260924 07:14:00.617842 23110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:14:00.619267 23110 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40511 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:14:00.622081 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 23042
I20260924 07:14:00.622200 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-1/wal/instance
I20260924 07:14:00.622380 392 external_mini_cluster.cc:1428] Setting key 2507f6547d302e2bc141dee584edadb8
Sep 24 07:14:00 dist-test-slave-14ww krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234040, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:14:00.628662 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.619872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55026 (local address 127.0.98.62:40511)
0924 07:14:00.619933 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:00.619935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:00.620127 (+ 192us) server_negotiation.cc:407] Connection header received
0924 07:14:00.620579 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:00.620580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:00.620591 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:00.620647 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:00.621259 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.621728 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.622824 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.623158 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.623232 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:00.625838 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:14:00.625840 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:14:00.625840 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:14:00.625849 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:14:00.627253 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:00.627658 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:00.627660 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:00.627660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:00.627720 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:00.627979 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:00.627980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:00.627980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:00.628091 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:14:00.628164 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:00.628467 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 07:14:00.628572 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 07:14:00.629107 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.619861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40511 (local address 127.0.0.1:55026)
0924 07:14:00.620003 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:00.620042 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 07:14:00.620491 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:14:00.620716 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:14:00.620720 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:14:00.621000 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:14:00.621177 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:00.621181 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.621804 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.621805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:00.622739 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:00.622742 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.623263 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.623264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:00.623379 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:00.623884 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:14:00.623894 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:14:00.625740 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:14:00.627352 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:14:00.627360 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:14:00.627361 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:14:00.627581 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:14:00.627787 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:14:00.627788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:14:00.627789 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:14:00.627902 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:14:00.628157 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:14:00.628163 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:14:00.628298 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 07:14:00.628566 (+ 268us) client_negotiation.cc:240] Negotiation successful
0924 07:14:00.628750 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
I20260924 07:14:00.629716 23110 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42243 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 07:14:00 dist-test-slave-14ww krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234040, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 07:14:00.636946 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.629966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42243 (local address 127.0.0.1:57558)
0924 07:14:00.630018 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:00.630029 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 07:14:00.630068 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:14:00.630320 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:14:00.630321 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:14:00.630465 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:14:00.630506 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:00.630508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.631381 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.631382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:00.631857 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:00.631859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.632429 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.632430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:00.632485 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:00.632999 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:14:00.633004 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:14:00.634131 (+ 1127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:14:00.635891 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:14:00.635892 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:14:00.635892 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:14:00.636060 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:14:00.636276 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:14:00.636277 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:14:00.636277 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:14:00.636330 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:14:00.636616 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:14:00.636617 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:14:00.636654 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 07:14:00.636718 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 07:14:00.636832 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 07:14:00.637008 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.630058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57558 (local address 127.0.98.61:42243)
0924 07:14:00.630188 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:00.630192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:00.630205 (+ 13us) server_negotiation.cc:407] Connection header received
0924 07:14:00.630220 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:00.630221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:00.630252 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:00.630319 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:00.630589 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.631307 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.631942 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.632353 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.632418 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:00.634233 (+ 1815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:14:00.634240 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:14:00.634241 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:14:00.634247 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:14:00.635801 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:00.636153 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:00.636156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:00.636157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:00.636210 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:00.636412 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:00.636413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:00.636413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:00.636544 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:14:00.636637 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:00.636769 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 07:14:00.636885 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":111,"thread_start_us":89,"threads_started":1}
I20260924 07:14:00.638015 23110 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37023 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:14:00.638451 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.638333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37023 (local address 127.0.0.1:56372)
0924 07:14:00.638383 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:00.638428 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 07:14:00.638618 23058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37023: connect: Connection refused (error 111)
I20260924 07:14:00.639031 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.638946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37023 (local address 127.0.0.1:56380)
0924 07:14:00.638994 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:00.639016 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 07:14:00.639150 23110 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37023: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37023: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 07:14:00.648020 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIclK1b/build/release/bin/kudu
/tmp/dist-test-taskIclK1b/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37023
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40511,127.0.98.61:42243,127.0.98.60:37023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 07:14:00.688372 23110 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37023 } attempt: 1
I20260924 07:14:00.688961 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.688795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37023 (local address 127.0.0.1:56394)
0924 07:14:00.688876 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:00.688921 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 07:14:00.689374 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.689287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37023 (local address 127.0.0.1:56400)
0924 07:14:00.689328 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:00.689354 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 07:14:00.689502 23110 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37023: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37023: connect: Connection refused (error 111)
I20260924 07:14:00.692824 23035 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42243 } attempt: 3
Sep 24 07:14:00 dist-test-slave-14ww krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234040, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 07:14:00.699879 23037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.693106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42243 (local address 127.0.0.1:57562)
0924 07:14:00.693162 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:00.693174 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:14:00.693226 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:14:00.693398 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:14:00.693398 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:14:00.693560 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:14:00.693623 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:00.693626 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.694308 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.694310 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:00.694834 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:00.694835 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.695242 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.695243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:00.695322 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:00.695826 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:14:00.695831 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:14:00.697145 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:14:00.698982 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:14:00.698983 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:14:00.698983 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:14:00.699146 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:14:00.699342 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:14:00.699342 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:14:00.699343 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:14:00.699399 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:14:00.699652 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:14:00.699653 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:14:00.699690 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 07:14:00.699742 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 07:14:00.699814 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 07:14:00.699951 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.693155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57562 (local address 127.0.98.61:42243)
0924 07:14:00.693196 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:00.693198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:00.693212 (+ 14us) server_negotiation.cc:407] Connection header received
0924 07:14:00.693308 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:00.693309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:00.693319 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:00.693391 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:00.693713 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.694212 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.694924 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.695156 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.695231 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:00.697251 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:14:00.697252 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:14:00.697252 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:14:00.697260 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:14:00.698885 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:00.699212 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:00.699214 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:00.699214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:00.699274 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:00.699458 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:00.699459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:00.699459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:00.699589 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:14:00.699656 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:00.699780 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 07:14:00.699856 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 07:14:00.700363 23035 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37023 } has no permanent_uuid. Determining permanent_uuid...
I20260924 07:14:00.700718 23037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.700593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37023 (local address 127.0.0.1:56414)
0924 07:14:00.700649 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:00.700683 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 07:14:00.701043 23037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.700956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37023 (local address 127.0.0.1:56418)
0924 07:14:00.700998 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:00.701023 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 07:14:00.701144 23035 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37023: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37023: connect: Connection refused (error 111)
W20260924 07:14:00.729811 23117 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 07:14:00.730017 23117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 07:14:00.730036 23117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 07:14:00.730055 23117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 07:14:00.731825 23117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 07:14:00.731870 23117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 07:14:00.731885 23117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 07:14:00.731899 23117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 07:14:00.731910 23117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 07:14:00.732301 23035 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37023 } attempt: 1
I20260924 07:14:00.732784 23037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.732620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37023 (local address 127.0.0.1:56422)
0924 07:14:00.732684 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:00.732747 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 07:14:00.733156 23037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.733056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37023 (local address 127.0.0.1:56436)
0924 07:14:00.733102 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:00.733132 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 07:14:00.733259 23035 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37023: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37023: connect: Connection refused (error 111)
I20260924 07:14:00.733757 23117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40511,127.0.98.61:42243,127.0.98.60:37023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 06:13:45 UTC on ccycloud.quasar-lsaogh.root.comops.site
build id 4669408
I20260924 07:14:00.733937 23117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 07:14:00.734092 23117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 07:14:00.738737 23122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 07:14:00.738814 23123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:14:00.738874 23117 server_base.cc:1034] running on GCE node
W20260924 07:14:00.738736 23125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 07:14:00.739238 23117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 07:14:00.739519 23117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 07:14:00.740676 23117 hybrid_clock.cc:648] HybridClock initialized: now 1790234040740672 us; error 36 us; skew 500 ppm
Sep 24 07:14:00 dist-test-slave-14ww krb5kdc[22954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234040, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 07:14:00.743145 23117 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 07:14:00.743609 23117 webserver.cc:460] Webserver started at http://127.0.98.60:41059/ using document root <none> and password file <none>
I20260924 07:14:00.743824 23117 fs_manager.cc:359] Metadata directory not provided
I20260924 07:14:00.743889 23117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 07:14:00.744041 23117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 07:14:00.745100 23117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-2/data/instance:
uuid: "25f63d0a964f4ba6a1f9ef53f4f36ee9"
format_stamp: "Formatted at 2026-09-24 07:14:00 on dist-test-slave-14ww"
server_key: "36eb8c54621947c3edd737928c70e5f5"
server_key_iv: "8757f17633e403f72cc68d38b4244026"
server_key_version: "encryptionkey@0"
I20260924 07:14:00.745440 23117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-2/wal/instance:
uuid: "25f63d0a964f4ba6a1f9ef53f4f36ee9"
format_stamp: "Formatted at 2026-09-24 07:14:00 on dist-test-slave-14ww"
server_key: "36eb8c54621947c3edd737928c70e5f5"
server_key_iv: "8757f17633e403f72cc68d38b4244026"
server_key_version: "encryptionkey@0"
I20260924 07:14:00.746683 23117 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 07:14:00.747418 23132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 07:14:00.747608 23117 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 07:14:00.747701 23117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-2/wal
uuid: "25f63d0a964f4ba6a1f9ef53f4f36ee9"
format_stamp: "Formatted at 2026-09-24 07:14:00 on dist-test-slave-14ww"
server_key: "36eb8c54621947c3edd737928c70e5f5"
server_key_iv: "8757f17633e403f72cc68d38b4244026"
server_key_version: "encryptionkey@0"
I20260924 07:14:00.747782 23117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 07:14:00.757645 23110 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37023 } attempt: 2
I20260924 07:14:00.757782 23117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 07:14:00.758090 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.757940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37023 (local address 127.0.0.1:56448)
0924 07:14:00.757994 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:00.758052 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 07:14:00.758526 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.758438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37023 (local address 127.0.0.1:56452)
0924 07:14:00.758481 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:00.758507 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 07:14:00.758662 23110 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37023: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37023: connect: Connection refused (error 111)
I20260924 07:14:00.759804 23117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 07:14:00.759956 23117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 07:14:00.763993 23117 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37023
I20260924 07:14:00.764024 23184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37023 every 8 connection(s)
I20260924 07:14:00.764339 23117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-2/data/info.pb
I20260924 07:14:00.765347 23185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 07:14:00.766781 23185 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40511 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:14:00 dist-test-slave-14ww krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234040, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:14:00.772571 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIclK1b/build/release/bin/kudu as pid 23117
I20260924 07:14:00.772691 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIclK1b/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790233772407728-392-0/minicluster-data/master-2/wal/instance
I20260924 07:14:00.772912 392 external_mini_cluster.cc:1428] Setting key 1cc1a67e48336de9c7fd1db8a65acfdf
I20260924 07:14:00.775597 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.767453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55030 (local address 127.0.98.62:40511)
0924 07:14:00.767510 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:00.767512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:00.767690 (+ 178us) server_negotiation.cc:407] Connection header received
0924 07:14:00.768118 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:00.768119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:00.768129 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:00.768178 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:00.768711 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.769218 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.770219 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.770392 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.770444 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:00.772577 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:14:00.772579 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:14:00.772579 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:14:00.772587 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:14:00.774145 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:00.774664 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:00.774665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:00.774665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:00.774725 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:00.774951 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:00.774952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:00.774953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:00.775077 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:14:00.775144 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:00.775439 (+ 295us) server_negotiation.cc:299] Negotiation successful
0924 07:14:00.775527 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 07:14:00.775799 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.767425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40511 (local address 127.0.0.1:55030)
0924 07:14:00.767568 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:00.767607 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 07:14:00.768042 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:14:00.768235 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:14:00.768237 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:14:00.768470 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:14:00.768644 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:00.768647 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.769357 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.769358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:00.770143 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:00.770146 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.770480 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.770481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:00.770573 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:00.771065 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:14:00.771073 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:14:00.772484 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:14:00.774224 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:14:00.774231 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:14:00.774234 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:14:00.774596 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:14:00.774794 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:14:00.774795 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:14:00.774795 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:14:00.774891 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:14:00.775168 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:14:00.775174 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:14:00.775285 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 07:14:00.775415 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 07:14:00.775587 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":76,"threads_started":1}
I20260924 07:14:00.776394 23185 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42243 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:14:00 dist-test-slave-14ww krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 07:14:00 dist-test-slave-14ww krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234040, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 07:14:00.782061 23189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.773311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56468 (local address 127.0.98.60:37023)
0924 07:14:00.773481 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:00.773484 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:00.773716 (+ 232us) server_negotiation.cc:407] Connection header received
0924 07:14:00.773743 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:00.773746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:00.773759 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:00.773846 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:00.774509 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.775448 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.776456 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.776873 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.776951 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:00.779190 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:14:00.779198 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:14:00.779198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:14:00.779204 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:14:00.780788 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:00.781193 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:00.781195 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:00.781196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:00.781246 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:00.781479 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:00.781480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:00.781480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:00.781619 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:14:00.781715 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:00.781841 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 07:14:00.781946 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":75,"threads_started":1}
I20260924 07:14:00.783259 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.776711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57576 (local address 127.0.98.61:42243)
0924 07:14:00.776770 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:00.776773 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:00.776782 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:14:00.776842 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:00.776843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:00.776853 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:00.776909 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:00.777172 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.777614 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.778275 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.778490 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.778576 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:00.780488 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:14:00.780489 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:14:00.780490 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:14:00.780497 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:14:00.782122 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:00.782464 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:00.782465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:00.782466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:00.782528 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:00.782738 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:00.782739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:00.782740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:00.782885 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:14:00.782953 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:00.783127 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 07:14:00.783200 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 07:14:00.783280 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.776646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42243 (local address 127.0.0.1:57576)
0924 07:14:00.776718 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:00.776743 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 07:14:00.776781 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:14:00.776915 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:14:00.776915 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:14:00.777047 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:14:00.777087 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:00.777088 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.777705 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.777707 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:00.778190 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:00.778192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.778577 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.778578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:00.778648 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:00.779044 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:14:00.779048 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:14:00.780401 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:14:00.782204 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:14:00.782205 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:14:00.782205 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:14:00.782395 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:14:00.782602 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:14:00.782603 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:14:00.782603 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:14:00.782671 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:14:00.782971 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:14:00.782972 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:14:00.783015 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 07:14:00.783085 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 07:14:00.783168 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 07:14:00.783581 23185 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37023 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 07:14:00 dist-test-slave-14ww krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234040, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 07:14:00 dist-test-slave-14ww krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 07:14:00.790835 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.783803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37023 (local address 127.0.0.1:56472)
0924 07:14:00.783855 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:00.783867 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 07:14:00.783901 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:14:00.784023 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:14:00.784023 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:14:00.784119 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:14:00.784142 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:00.784143 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.784937 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.784938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:00.785577 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:00.785579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.786124 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.786125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:00.786182 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:00.786474 (+ 292us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:14:00.786479 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:14:00.788207 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:14:00.789882 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:14:00.789883 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:14:00.789883 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:14:00.790040 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:14:00.790243 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:14:00.790244 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:14:00.790245 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:14:00.790295 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:14:00.790539 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:14:00.790539 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:14:00.790574 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 07:14:00.790641 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 07:14:00.790733 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 07:14:00.790884 23189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.783898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56472 (local address 127.0.98.60:37023)
0924 07:14:00.783932 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:00.783933 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:00.783940 (+ 7us) server_negotiation.cc:407] Connection header received
0924 07:14:00.783957 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:00.783958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:00.783967 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:00.784022 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:00.784205 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.784865 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.785663 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.786050 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.786114 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:00.788304 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:14:00.788305 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:14:00.788306 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:14:00.788312 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:14:00.789778 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:00.790125 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:00.790126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:00.790126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:00.790179 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:00.790374 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:00.790375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:00.790375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:00.790488 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:14:00.790546 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:00.790670 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 07:14:00.790778 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":18}
I20260924 07:14:00.791415 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.782841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55042 (local address 127.0.98.62:40511)
0924 07:14:00.782875 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:00.782878 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:00.782936 (+ 58us) server_negotiation.cc:407] Connection header received
0924 07:14:00.783004 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:00.783005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:00.783014 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:00.783048 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:00.783809 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.784479 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.785307 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.785489 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.785540 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:00.788390 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:14:00.788392 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:14:00.788392 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:14:00.788399 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:14:00.789918 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:00.790267 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:00.790268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:00.790269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:00.790323 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:00.790540 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:00.790541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:00.790541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:00.790646 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:14:00.790818 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:00.790852 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 07:14:00.791326 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 07:14:00.792377 23185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25f63d0a964f4ba6a1f9ef53f4f36ee9: Bootstrap starting.
I20260924 07:14:00.793292 23185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25f63d0a964f4ba6a1f9ef53f4f36ee9: Neither blocks nor log segments found. Creating new log.
I20260924 07:14:00.793570 23185 log.cc:824] T 00000000000000000000000000000000 P 25f63d0a964f4ba6a1f9ef53f4f36ee9: Log is configured to *not* fsync() on all Append() calls
I20260924 07:14:00.794246 23185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25f63d0a964f4ba6a1f9ef53f4f36ee9: No bootstrap required, opened a new log
Sep 24 07:14:00 dist-test-slave-14ww krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 07:14:00.795967 23185 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25f63d0a964f4ba6a1f9ef53f4f36ee9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5109dea49c44314a9d2ca79d9f1432b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40511 } } peers { permanent_uuid: "5559dfcea0fe4c4992d0aac393e77f02" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42243 } } peers { permanent_uuid: "25f63d0a964f4ba6a1f9ef53f4f36ee9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37023 } }
I20260924 07:14:00.796103 23185 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25f63d0a964f4ba6a1f9ef53f4f36ee9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25f63d0a964f4ba6a1f9ef53f4f36ee9, State: Initialized, Role: FOLLOWER
I20260924 07:14:00.796187 23185 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25f63d0a964f4ba6a1f9ef53f4f36ee9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5109dea49c44314a9d2ca79d9f1432b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40511 } } peers { permanent_uuid: "5559dfcea0fe4c4992d0aac393e77f02" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42243 } } peers { permanent_uuid: "25f63d0a964f4ba6a1f9ef53f4f36ee9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37023 } }
I20260924 07:14:00.796494 23185 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25f63d0a964f4ba6a1f9ef53f4f36ee9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:14:00.796484 23191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25f63d0a964f4ba6a1f9ef53f4f36ee9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5109dea49c44314a9d2ca79d9f1432b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40511 } } peers { permanent_uuid: "5559dfcea0fe4c4992d0aac393e77f02" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42243 } } peers { permanent_uuid: "25f63d0a964f4ba6a1f9ef53f4f36ee9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37023 } } }
I20260924 07:14:00.796656 23191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25f63d0a964f4ba6a1f9ef53f4f36ee9 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 07:14:00.798781 23202 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 25f63d0a964f4ba6a1f9ef53f4f36ee9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:14:00.798841 23202 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:14:00.799736 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.792037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57584 (local address 127.0.98.61:42243)
0924 07:14:00.792071 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:00.792073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:00.792080 (+ 7us) server_negotiation.cc:407] Connection header received
0924 07:14:00.792136 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:00.792137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:00.792145 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:00.792192 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:00.792699 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.793194 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.793968 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.794171 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.794254 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:00.796674 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:14:00.796675 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:14:00.796676 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:14:00.796683 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:14:00.798581 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:00.798929 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:00.798930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:00.798931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:00.798999 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:00.799218 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:00.799219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:00.799219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:00.799355 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:14:00.799516 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:00.799563 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 07:14:00.799639 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 07:14:00.801095 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 07:14:00.798941 23117 master_runner.cc:186] Error getting master registration for 127.0.98.62:40511: OK, instance_id { permanent_uuid: "b5109dea49c44314a9d2ca79d9f1432b" instance_seqno: 1790234040465899 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 07:14:00.801788 23117 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 07:14:00.802248 23009 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b5109dea49c44314a9d2ca79d9f1432b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55042
I20260924 07:14:00.808380 23035 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37023 } attempt: 2
Sep 24 07:14:00 dist-test-slave-14ww krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234040, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 07:14:00.815415 23037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.808638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37023 (local address 127.0.0.1:56476)
0924 07:14:00.808690 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:00.808703 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 07:14:00.808796 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:14:00.808951 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:14:00.808952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:14:00.809103 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:14:00.809141 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:00.809142 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.809759 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.809760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:00.810245 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:00.810247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.810606 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.810607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:00.810667 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:00.811107 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:14:00.811111 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:14:00.812483 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:14:00.814292 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:14:00.814293 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:14:00.814294 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:14:00.814513 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:14:00.814723 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:14:00.814724 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:14:00.814724 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:14:00.814784 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:14:00.815094 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:14:00.815095 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:14:00.815141 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 07:14:00.815207 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 07:14:00.815303 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 07:14:00.815423 23189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.808696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56476 (local address 127.0.98.60:37023)
0924 07:14:00.808771 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:00.808773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:00.808781 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:14:00.808857 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:00.808858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:00.808868 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:00.808935 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:00.809215 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.809679 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.810331 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.810522 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.810605 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:00.812581 (+ 1976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:14:00.812582 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:14:00.812582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:14:00.812590 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:14:00.814205 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:00.814592 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:00.814593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:00.814594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:00.814654 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:00.814849 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:00.814850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:00.814850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:00.815028 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:14:00.815098 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:00.815252 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 07:14:00.815331 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 07:14:00.817057 23035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5109dea49c44314a9d2ca79d9f1432b: Bootstrap starting.
I20260924 07:14:00.817775 23035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5109dea49c44314a9d2ca79d9f1432b: Neither blocks nor log segments found. Creating new log.
I20260924 07:14:00.818068 23035 log.cc:824] T 00000000000000000000000000000000 P b5109dea49c44314a9d2ca79d9f1432b: Log is configured to *not* fsync() on all Append() calls
I20260924 07:14:00.818816 23035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5109dea49c44314a9d2ca79d9f1432b: No bootstrap required, opened a new log
I20260924 07:14:00.819821 23035 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5109dea49c44314a9d2ca79d9f1432b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5109dea49c44314a9d2ca79d9f1432b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40511 } } peers { permanent_uuid: "5559dfcea0fe4c4992d0aac393e77f02" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42243 } } peers { permanent_uuid: "25f63d0a964f4ba6a1f9ef53f4f36ee9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37023 } }
I20260924 07:14:00.819962 23035 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5109dea49c44314a9d2ca79d9f1432b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5109dea49c44314a9d2ca79d9f1432b, State: Initialized, Role: FOLLOWER
I20260924 07:14:00.820056 23035 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5109dea49c44314a9d2ca79d9f1432b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5109dea49c44314a9d2ca79d9f1432b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40511 } } peers { permanent_uuid: "5559dfcea0fe4c4992d0aac393e77f02" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42243 } } peers { permanent_uuid: "25f63d0a964f4ba6a1f9ef53f4f36ee9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37023 } }
I20260924 07:14:00.820344 23205 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5109dea49c44314a9d2ca79d9f1432b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5109dea49c44314a9d2ca79d9f1432b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40511 } } peers { permanent_uuid: "5559dfcea0fe4c4992d0aac393e77f02" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42243 } } peers { permanent_uuid: "25f63d0a964f4ba6a1f9ef53f4f36ee9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37023 } } }
I20260924 07:14:00.820389 23035 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5109dea49c44314a9d2ca79d9f1432b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:14:00.820453 23009 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5109dea49c44314a9d2ca79d9f1432b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 07:14:00.820518 23009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5109dea49c44314a9d2ca79d9f1432b [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:14:00.821353 23009 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5109dea49c44314a9d2ca79d9f1432b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5109dea49c44314a9d2ca79d9f1432b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40511 } } peers { permanent_uuid: "5559dfcea0fe4c4992d0aac393e77f02" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42243 } } peers { permanent_uuid: "25f63d0a964f4ba6a1f9ef53f4f36ee9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37023 } }
I20260924 07:14:00.821628 23009 leader_election.cc:288] T 00000000000000000000000000000000 P b5109dea49c44314a9d2ca79d9f1432b [CANDIDATE]: Term 1 election: Requested vote from peers 5559dfcea0fe4c4992d0aac393e77f02 (127.0.98.61:42243), 25f63d0a964f4ba6a1f9ef53f4f36ee9 (127.0.98.60:37023)
I20260924 07:14:00.820439 23205 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5109dea49c44314a9d2ca79d9f1432b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:14:00.822141 23159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b5109dea49c44314a9d2ca79d9f1432b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "25f63d0a964f4ba6a1f9ef53f4f36ee9"
I20260924 07:14:00.822261 23159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25f63d0a964f4ba6a1f9ef53f4f36ee9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:14:00.823184 23159 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 25f63d0a964f4ba6a1f9ef53f4f36ee9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5109dea49c44314a9d2ca79d9f1432b in term 1.
I20260924 07:14:00.823407 22983 leader_election.cc:302] T 00000000000000000000000000000000 P b5109dea49c44314a9d2ca79d9f1432b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25f63d0a964f4ba6a1f9ef53f4f36ee9, b5109dea49c44314a9d2ca79d9f1432b; no voters:
I20260924 07:14:00.823640 23205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5109dea49c44314a9d2ca79d9f1432b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 07:14:00.824221 23205 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5109dea49c44314a9d2ca79d9f1432b [term 1 LEADER]: Becoming Leader. State: Replica: b5109dea49c44314a9d2ca79d9f1432b, State: Running, Role: LEADER
I20260924 07:14:00.823642 22967 master_runner.cc:186] Error getting master registration for 127.0.98.61:42243: OK, instance_id { permanent_uuid: "5559dfcea0fe4c4992d0aac393e77f02" instance_seqno: 1790234040615443 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 07:14:00.827042 22967 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 07:14:00.827198 23205 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5109dea49c44314a9d2ca79d9f1432b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5109dea49c44314a9d2ca79d9f1432b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40511 } } peers { permanent_uuid: "5559dfcea0fe4c4992d0aac393e77f02" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42243 } } peers { permanent_uuid: "25f63d0a964f4ba6a1f9ef53f4f36ee9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37023 } }
I20260924 07:14:00.827718 23205 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5109dea49c44314a9d2ca79d9f1432b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5109dea49c44314a9d2ca79d9f1432b. Latest consensus state: current_term: 1 leader_uuid: "b5109dea49c44314a9d2ca79d9f1432b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5109dea49c44314a9d2ca79d9f1432b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40511 } } peers { permanent_uuid: "5559dfcea0fe4c4992d0aac393e77f02" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42243 } } peers { permanent_uuid: "25f63d0a964f4ba6a1f9ef53f4f36ee9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37023 } } }
I20260924 07:14:00.827813 23205 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5109dea49c44314a9d2ca79d9f1432b [sys.catalog]: This master's current role is: LEADER
W20260924 07:14:00.828369 23217 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b5109dea49c44314a9d2ca79d9f1432b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 07:14:00.828423 23217 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 07:14:00.832475 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.823149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55056 (local address 127.0.98.62:40511)
0924 07:14:00.823186 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:00.823188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:00.823860 (+ 672us) server_negotiation.cc:407] Connection header received
0924 07:14:00.823952 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:00.823953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:00.823963 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:00.823988 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:00.824753 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.825473 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.826553 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.826805 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.826848 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:00.829270 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:14:00.829271 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:14:00.829271 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:14:00.829288 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:14:00.830835 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:00.831410 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:00.831411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:00.831411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:00.831477 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:00.831783 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:00.831784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:00.831784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:00.831918 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:14:00.831987 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:00.832298 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 07:14:00.832374 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 07:14:00.832988 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.822798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57590 (local address 127.0.98.61:42243)
0924 07:14:00.822839 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:00.822841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:00.823691 (+ 850us) server_negotiation.cc:407] Connection header received
0924 07:14:00.823849 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:00.823850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:00.823861 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:00.823891 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:00.824720 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.825485 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.826582 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.826897 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.826949 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:00.829017 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:14:00.829018 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:14:00.829018 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:14:00.829026 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:14:00.831125 (+ 2099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:00.831507 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:00.831508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:00.831508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:00.831561 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:00.831755 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:00.831756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:00.831756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:00.831877 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:14:00.832347 (+ 470us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:00.832801 (+ 454us) server_negotiation.cc:299] Negotiation successful
0924 07:14:00.832883 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 07:14:00.833323 23226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 07:14:00.833540 23226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 07:14:00.835438 23159 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 25f63d0a964f4ba6a1f9ef53f4f36ee9 [term 1 FOLLOWER]: Refusing update from remote peer b5109dea49c44314a9d2ca79d9f1432b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:14:00.835947 23205 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b5109dea49c44314a9d2ca79d9f1432b [LEADER]: Connected to new peer: Peer: permanent_uuid: "25f63d0a964f4ba6a1f9ef53f4f36ee9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:14:00.838497 23191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25f63d0a964f4ba6a1f9ef53f4f36ee9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5109dea49c44314a9d2ca79d9f1432b. Latest consensus state: current_term: 1 leader_uuid: "b5109dea49c44314a9d2ca79d9f1432b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5109dea49c44314a9d2ca79d9f1432b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40511 } } peers { permanent_uuid: "5559dfcea0fe4c4992d0aac393e77f02" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42243 } } peers { permanent_uuid: "25f63d0a964f4ba6a1f9ef53f4f36ee9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37023 } } }
I20260924 07:14:00.838598 23191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25f63d0a964f4ba6a1f9ef53f4f36ee9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:14:00.841133 23189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.823579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56484 (local address 127.0.98.60:37023)
0924 07:14:00.823614 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:00.823615 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:00.833345 (+ 9730us) server_negotiation.cc:407] Connection header received
0924 07:14:00.833507 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:00.833508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:00.833519 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:00.833563 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:00.834367 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.835084 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.836033 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.836382 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.836432 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:00.837872 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:14:00.837873 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:14:00.837874 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:14:00.837882 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:14:00.839632 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:00.840254 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:00.840255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:00.840255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:00.840316 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:00.840564 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:00.840565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:00.840565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:00.840703 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:14:00.840834 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:00.840976 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 07:14:00.841052 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 07:14:00.842128 23205 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5109dea49c44314a9d2ca79d9f1432b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b5109dea49c44314a9d2ca79d9f1432b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5109dea49c44314a9d2ca79d9f1432b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40511 } } peers { permanent_uuid: "5559dfcea0fe4c4992d0aac393e77f02" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42243 } } peers { permanent_uuid: "25f63d0a964f4ba6a1f9ef53f4f36ee9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37023 } } }
I20260924 07:14:00.842208 23205 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5109dea49c44314a9d2ca79d9f1432b [sys.catalog]: This master's current role is: LEADER
I20260924 07:14:00.842198 23191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25f63d0a964f4ba6a1f9ef53f4f36ee9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b5109dea49c44314a9d2ca79d9f1432b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5109dea49c44314a9d2ca79d9f1432b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40511 } } peers { permanent_uuid: "5559dfcea0fe4c4992d0aac393e77f02" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42243 } } peers { permanent_uuid: "25f63d0a964f4ba6a1f9ef53f4f36ee9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37023 } } }
I20260924 07:14:00.842274 23191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25f63d0a964f4ba6a1f9ef53f4f36ee9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:14:00.842468 23228 mvcc.cc:205] Tried to move back new op lower bound from 7332798631258718208 to 7332798631229468672. Current Snapshot: MvccSnapshot[applied={T|T < 7332798631258718208}]
I20260924 07:14:00.843146 23226 catalog_manager.cc:1347] Generated new cluster ID: c2b3638100bf40f0875e1abebb82008a
I20260924 07:14:00.843194 23226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 07:14:00.847563 23229 mvcc.cc:205] Tried to move back new op lower bound from 7332798631258718208 to 7332798631229468672. Current Snapshot: MvccSnapshot[applied={T|T < 7332798631258718208 or (T in {7332798631258718208})}]
I20260924 07:14:00.849753 23110 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37023 } attempt: 3
Sep 24 07:14:00 dist-test-slave-14ww krb5kdc[22954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790234040, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 07:14:00.859302 23226 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 07:14:00.859537 23112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.849960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37023 (local address 127.0.0.1:56494)
0924 07:14:00.850028 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0924 07:14:00.850050 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 07:14:00.850081 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 07:14:00.850380 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 07:14:00.850381 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 07:14:00.850538 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 07:14:00.850579 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:00.850580 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.851527 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.851527 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:00.852283 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 07:14:00.852284 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.853490 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.853491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 07:14:00.853559 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:00.854042 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 07:14:00.854047 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 07:14:00.855430 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 07:14:00.858075 (+ 2645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:14:00.858076 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:14:00.858077 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:14:00.858306 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:14:00.858662 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 07:14:00.858663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 07:14:00.858664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 07:14:00.858744 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 07:14:00.859241 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 07:14:00.859242 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 07:14:00.859303 (+ 61us) client_negotiation.cc:769] Sending connection context
0924 07:14:00.859357 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 07:14:00.859446 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 07:14:00.860028 23226 catalog_manager.cc:1504] Loading token signing keys...
I20260924 07:14:00.860394 23189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.850138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56494 (local address 127.0.98.60:37023)
0924 07:14:00.850256 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:00.850258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:00.850264 (+ 6us) server_negotiation.cc:407] Connection header received
0924 07:14:00.850294 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:00.850294 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:00.850303 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:00.850343 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:00.850660 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.851386 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.852394 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.853375 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.853426 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:00.855529 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 07:14:00.855530 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 07:14:00.855531 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 07:14:00.855537 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 07:14:00.857914 (+ 2377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:00.858447 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:00.858448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:00.858448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:00.858539 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 07:14:00.858834 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 07:14:00.858835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 07:14:00.858835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 07:14:00.859099 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 07:14:00.859952 (+ 853us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:00.860024 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 07:14:00.860225 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260924 07:14:00.862010 23110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5559dfcea0fe4c4992d0aac393e77f02: Bootstrap starting.
I20260924 07:14:00.862660 23110 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5559dfcea0fe4c4992d0aac393e77f02: Neither blocks nor log segments found. Creating new log.
I20260924 07:14:00.862907 23110 log.cc:824] T 00000000000000000000000000000000 P 5559dfcea0fe4c4992d0aac393e77f02: Log is configured to *not* fsync() on all Append() calls
I20260924 07:14:00.867990 23110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5559dfcea0fe4c4992d0aac393e77f02: No bootstrap required, opened a new log
I20260924 07:14:00.869704 23110 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5559dfcea0fe4c4992d0aac393e77f02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5109dea49c44314a9d2ca79d9f1432b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40511 } } peers { permanent_uuid: "5559dfcea0fe4c4992d0aac393e77f02" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42243 } } peers { permanent_uuid: "25f63d0a964f4ba6a1f9ef53f4f36ee9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37023 } }
I20260924 07:14:00.869859 23110 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5559dfcea0fe4c4992d0aac393e77f02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5559dfcea0fe4c4992d0aac393e77f02, State: Initialized, Role: FOLLOWER
I20260924 07:14:00.869956 23110 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5559dfcea0fe4c4992d0aac393e77f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5109dea49c44314a9d2ca79d9f1432b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40511 } } peers { permanent_uuid: "5559dfcea0fe4c4992d0aac393e77f02" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42243 } } peers { permanent_uuid: "25f63d0a964f4ba6a1f9ef53f4f36ee9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37023 } }
I20260924 07:14:00.870227 23235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5559dfcea0fe4c4992d0aac393e77f02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5109dea49c44314a9d2ca79d9f1432b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40511 } } peers { permanent_uuid: "5559dfcea0fe4c4992d0aac393e77f02" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42243 } } peers { permanent_uuid: "25f63d0a964f4ba6a1f9ef53f4f36ee9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37023 } } }
I20260924 07:14:00.870384 23235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5559dfcea0fe4c4992d0aac393e77f02 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:14:00.870608 23084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b5109dea49c44314a9d2ca79d9f1432b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5559dfcea0fe4c4992d0aac393e77f02"
I20260924 07:14:00.870714 23084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5559dfcea0fe4c4992d0aac393e77f02 [term 0 FOLLOWER]: Advancing to term 1
I20260924 07:14:00.871392 23226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5109dea49c44314a9d2ca79d9f1432b: Generated new TSK 0
I20260924 07:14:00.871601 23226 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 07:14:00.871592 23084 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5559dfcea0fe4c4992d0aac393e77f02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5109dea49c44314a9d2ca79d9f1432b in term 1.
I20260924 07:14:00.871966 23083 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5559dfcea0fe4c4992d0aac393e77f02 [term 1 FOLLOWER]: Refusing update from remote peer b5109dea49c44314a9d2ca79d9f1432b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 07:14:00.870306 23110 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5559dfcea0fe4c4992d0aac393e77f02 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 07:14:00.872248 23232 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b5109dea49c44314a9d2ca79d9f1432b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5559dfcea0fe4c4992d0aac393e77f02" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 07:14:00.874902 23235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5559dfcea0fe4c4992d0aac393e77f02 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b5109dea49c44314a9d2ca79d9f1432b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5109dea49c44314a9d2ca79d9f1432b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40511 } } peers { permanent_uuid: "5559dfcea0fe4c4992d0aac393e77f02" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42243 } } peers { permanent_uuid: "25f63d0a964f4ba6a1f9ef53f4f36ee9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37023 } } }
I20260924 07:14:00.875044 23235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5559dfcea0fe4c4992d0aac393e77f02 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:14:00.880571 23248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5559dfcea0fe4c4992d0aac393e77f02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5109dea49c44314a9d2ca79d9f1432b. Latest consensus state: current_term: 1 leader_uuid: "b5109dea49c44314a9d2ca79d9f1432b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5109dea49c44314a9d2ca79d9f1432b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40511 } } peers { permanent_uuid: "5559dfcea0fe4c4992d0aac393e77f02" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42243 } } peers { permanent_uuid: "25f63d0a964f4ba6a1f9ef53f4f36ee9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37023 } } }
I20260924 07:14:00.880663 23248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5559dfcea0fe4c4992d0aac393e77f02 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 07:14:00.882059 23252 catalog_manager.cc:1259] Loaded cluster ID: c2b3638100bf40f0875e1abebb82008a
I20260924 07:14:00.882105 23252 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5559dfcea0fe4c4992d0aac393e77f02: loading cluster ID for follower catalog manager: success
I20260924 07:14:00.883199 23252 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5559dfcea0fe4c4992d0aac393e77f02: acquiring CA information for follower catalog manager: success
I20260924 07:14:00.883504 23252 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5559dfcea0fe4c4992d0aac393e77f02: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 07:14:00.896330 23114 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:14:00.894692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57592 (local address 127.0.98.61:42243)
0924 07:14:00.894732 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:00.894734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:00.895604 (+ 870us) server_negotiation.cc:407] Connection header received
0924 07:14:00.895708 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:00.895709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:00.895720 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:00.895749 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:00.896253 (+ 504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57592: BlockingRecv error: recv got EOF from 127.0.0.1:57592 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 07:14:00.897130 23038 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:14:00.895070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55062 (local address 127.0.98.62:40511)
0924 07:14:00.895111 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:00.895113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:00.896588 (+ 1475us) server_negotiation.cc:407] Connection header received
0924 07:14:00.896686 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:00.896687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:00.896697 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:00.896726 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:00.897068 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55062: BlockingRecv error: recv got EOF from 127.0.0.1:55062 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 07:14:00.897927 23189 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:14:00.895356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56496 (local address 127.0.98.60:37023)
0924 07:14:00.895392 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:00.895394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:00.897378 (+ 1984us) server_negotiation.cc:407] Connection header received
0924 07:14:00.897468 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:00.897469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:00.897479 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:00.897526 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:00.897865 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56496: BlockingRecv error: recv got EOF from 127.0.0.1:56496 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 07:14:00.904806 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.901372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57600 (local address 127.0.98.61:42243)
0924 07:14:00.901403 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:00.901404 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:00.901602 (+ 198us) server_negotiation.cc:407] Connection header received
0924 07:14:00.901700 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:00.901701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:00.901734 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:00.901774 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:14:00.901979 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.902755 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.903882 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.904175 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.904213 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:00.904430 (+ 217us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:14:00.904543 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:14:00.904588 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:00.904695 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 07:14:00.904712 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
W20260924 07:14:00.907719 23189 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:14:00.907047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56506 (local address 127.0.98.60:37023)
0924 07:14:00.907081 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:00.907083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:00.907234 (+ 151us) server_negotiation.cc:407] Connection header received
0924 07:14:00.907338 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:00.907339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:00.907354 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:00.907382 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:00.907672 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56506: BlockingRecv error: recv got EOF from 127.0.0.1:56506 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 07:14:00.912968 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.909869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57616 (local address 127.0.98.61:42243)
0924 07:14:00.909900 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:00.909902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:00.910104 (+ 202us) server_negotiation.cc:407] Connection header received
0924 07:14:00.910191 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:00.910192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:00.910200 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:00.910225 (+ 25us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:14:00.910378 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.911099 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.912178 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.912458 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.912496 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:00.912688 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:14:00.912718 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:14:00.912768 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:00.912887 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 07:14:00.912904 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 07:14:00.915786 23189 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:14:00.915134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56512 (local address 127.0.98.60:37023)
0924 07:14:00.915167 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:00.915169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:00.915328 (+ 159us) server_negotiation.cc:407] Connection header received
0924 07:14:00.915418 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:00.915419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:00.915429 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:00.915457 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:00.915742 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56512: BlockingRecv error: recv got EOF from 127.0.0.1:56512 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 07:14:00.922094 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.918779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57622 (local address 127.0.98.61:42243)
0924 07:14:00.918810 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:00.918812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:00.919020 (+ 208us) server_negotiation.cc:407] Connection header received
0924 07:14:00.919106 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:00.919107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:00.919115 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:00.919141 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:14:00.919296 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.920008 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.921227 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.921539 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.921578 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:00.921801 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:14:00.921854 (+ 53us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:14:00.921883 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:00.922015 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 07:14:00.922031 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
W20260924 07:14:00.924726 23189 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:14:00.924086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56522 (local address 127.0.98.60:37023)
0924 07:14:00.924120 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:00.924122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:00.924272 (+ 150us) server_negotiation.cc:407] Connection header received
0924 07:14:00.924362 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:00.924363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:00.924376 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:00.924404 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:00.924683 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56522: BlockingRecv error: recv got EOF from 127.0.0.1:56522 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 07:14:00.932812 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.929776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57636 (local address 127.0.98.61:42243)
0924 07:14:00.929807 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:00.929808 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:00.930015 (+ 207us) server_negotiation.cc:407] Connection header received
0924 07:14:00.930106 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:00.930107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:00.930115 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:00.930140 (+ 25us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:14:00.930285 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.930991 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.932050 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.932331 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.932369 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:00.932560 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:14:00.932589 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:14:00.932614 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:00.932712 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 07:14:00.932727 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 07:14:00.935508 23189 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:14:00.934840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56536 (local address 127.0.98.60:37023)
0924 07:14:00.934874 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:00.934876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:00.935032 (+ 156us) server_negotiation.cc:407] Connection header received
0924 07:14:00.935124 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:00.935125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:00.935136 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:00.935165 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:00.935462 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56536: BlockingRecv error: recv got EOF from 127.0.0.1:56536 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 07:14:00.948647 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.944525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57638 (local address 127.0.98.61:42243)
0924 07:14:00.944558 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:00.944559 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:00.944782 (+ 223us) server_negotiation.cc:407] Connection header received
0924 07:14:00.944889 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:00.944890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:00.944900 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:00.944927 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:14:00.945081 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.945786 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.947900 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.948185 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.948221 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:00.948404 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:14:00.948434 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:14:00.948461 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:00.948561 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 07:14:00.948577 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 07:14:00.951704 23189 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:14:00.950834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56546 (local address 127.0.98.60:37023)
0924 07:14:00.950873 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:00.950874 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:00.951084 (+ 210us) server_negotiation.cc:407] Connection header received
0924 07:14:00.951175 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:00.951176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:00.951186 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:00.951214 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:00.951660 (+ 446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56546: BlockingRecv error: recv got EOF from 127.0.0.1:56546 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 07:14:00.972267 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:00.968941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57644 (local address 127.0.98.61:42243)
0924 07:14:00.968981 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:00.968983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:00.969233 (+ 250us) server_negotiation.cc:407] Connection header received
0924 07:14:00.969335 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:00.969336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:00.969347 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:00.969381 (+ 34us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:14:00.969564 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.970295 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.971376 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:00.971654 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:00.971698 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:00.971992 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:14:00.972029 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:14:00.972057 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:00.972168 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 07:14:00.972187 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 07:14:00.975670 23189 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:14:00.974877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56550 (local address 127.0.98.60:37023)
0924 07:14:00.974918 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:00.974920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:00.975123 (+ 203us) server_negotiation.cc:407] Connection header received
0924 07:14:00.975220 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:00.975221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:00.975232 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:00.975262 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:00.975618 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56550: BlockingRecv error: recv got EOF from 127.0.0.1:56550 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 07:14:01.012140 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:01.009067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57658 (local address 127.0.98.61:42243)
0924 07:14:01.009112 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:01.009115 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:01.009240 (+ 125us) server_negotiation.cc:407] Connection header received
0924 07:14:01.009281 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:01.009282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:01.009294 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:01.009377 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:14:01.009516 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:01.010320 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:01.011403 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:01.011679 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:01.011757 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:01.011898 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:14:01.011932 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:14:01.011989 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:01.012040 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 07:14:01.012060 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 07:14:01.014789 23189 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:14:01.014129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56560 (local address 127.0.98.60:37023)
0924 07:14:01.014178 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:01.014181 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:01.014283 (+ 102us) server_negotiation.cc:407] Connection header received
0924 07:14:01.014340 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:01.014341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:01.014351 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:01.014391 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:01.014734 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56560: BlockingRecv error: recv got EOF from 127.0.0.1:56560 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 07:14:01.082484 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:01.079986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57674 (local address 127.0.98.61:42243)
0924 07:14:01.080090 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:01.080093 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:01.080241 (+ 148us) server_negotiation.cc:407] Connection header received
0924 07:14:01.080332 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:01.080333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:01.080341 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:01.080392 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:14:01.080515 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:01.081033 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:01.081768 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:01.081951 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:01.082006 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:01.082207 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:14:01.082239 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:14:01.082292 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:01.082349 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 07:14:01.082365 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260924 07:14:01.084815 23189 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:14:01.084213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56572 (local address 127.0.98.60:37023)
0924 07:14:01.084290 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:01.084292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:01.084342 (+ 50us) server_negotiation.cc:407] Connection header received
0924 07:14:01.084372 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:01.084373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:01.084383 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:01.084430 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:01.084712 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56572: BlockingRecv error: recv got EOF from 127.0.0.1:56572 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 07:14:01.216346 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:01.213917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57678 (local address 127.0.98.61:42243)
0924 07:14:01.213993 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:01.213996 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:01.214112 (+ 116us) server_negotiation.cc:407] Connection header received
0924 07:14:01.214194 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:01.214199 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:01.214215 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:01.214285 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:14:01.214423 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:01.214898 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:01.215654 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:01.215845 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:01.215928 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:01.216074 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:14:01.216111 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:14:01.216171 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:01.216236 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 07:14:01.216255 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260924 07:14:01.218673 23189 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:14:01.217961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56578 (local address 127.0.98.60:37023)
0924 07:14:01.218032 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:01.218034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:01.218140 (+ 106us) server_negotiation.cc:407] Connection header received
0924 07:14:01.218207 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:01.218208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:01.218221 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:01.218324 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:01.218634 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56578: BlockingRecv error: recv got EOF from 127.0.0.1:56578 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 07:14:01.324988 23205 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5109dea49c44314a9d2ca79d9f1432b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b5109dea49c44314a9d2ca79d9f1432b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5109dea49c44314a9d2ca79d9f1432b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40511 } } peers { permanent_uuid: "5559dfcea0fe4c4992d0aac393e77f02" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42243 } } peers { permanent_uuid: "25f63d0a964f4ba6a1f9ef53f4f36ee9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37023 } } }
I20260924 07:14:01.325169 23205 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5109dea49c44314a9d2ca79d9f1432b [sys.catalog]: This master's current role is: LEADER
I20260924 07:14:01.478262 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:01.475928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57686 (local address 127.0.98.61:42243)
0924 07:14:01.475990 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:01.475992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:01.476048 (+ 56us) server_negotiation.cc:407] Connection header received
0924 07:14:01.476103 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:01.476103 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:01.476112 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:01.476163 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:14:01.476294 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:01.476791 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:01.477543 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:01.477722 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:01.477777 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:01.477968 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:14:01.478001 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:14:01.478055 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:01.478124 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 07:14:01.478141 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260924 07:14:01.480638 23189 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 07:14:01.480058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56590 (local address 127.0.98.60:37023)
0924 07:14:01.480125 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:01.480127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:01.480136 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:14:01.480220 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:01.480221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:01.480233 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:01.480282 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 07:14:01.480581 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56590: BlockingRecv error: recv got EOF from 127.0.0.1:56590 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 07:14:01.799270 23202 catalog_manager.cc:1259] Loaded cluster ID: c2b3638100bf40f0875e1abebb82008a
I20260924 07:14:01.799348 23202 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 25f63d0a964f4ba6a1f9ef53f4f36ee9: loading cluster ID for follower catalog manager: success
I20260924 07:14:01.800138 23202 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 25f63d0a964f4ba6a1f9ef53f4f36ee9: acquiring CA information for follower catalog manager: success
I20260924 07:14:01.800429 23202 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 25f63d0a964f4ba6a1f9ef53f4f36ee9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 07:14:01.996330 23368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:01.994021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57696 (local address 127.0.98.61:42243)
0924 07:14:01.994140 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:01.994142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:01.994210 (+ 68us) server_negotiation.cc:407] Connection header received
0924 07:14:01.994274 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:01.994275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:01.994286 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:01.994346 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:14:01.994455 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:01.994984 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:01.995749 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:01.995971 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:01.996023 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:01.996136 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:14:01.996163 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:14:01.996201 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:01.996247 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 07:14:01.996266 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":65,"threads_started":1}
I20260924 07:14:02.000191 23374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:01.998006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56592 (local address 127.0.98.60:37023)
0924 07:14:01.998103 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:01.998105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:01.998114 (+ 9us) server_negotiation.cc:407] Connection header received
0924 07:14:01.998177 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:01.998178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:01.998226 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:01.998331 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:14:01.998406 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:01.998861 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:01.999529 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:01.999712 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:01.999760 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:01.999877 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:14:01.999975 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:14:02.000030 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:02.000059 (+ 29us) server_negotiation.cc:299] Negotiation successful
0924 07:14:02.000074 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":57,"threads_started":1}
I20260924 07:14:02.004407 23368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:02.001689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57706 (local address 127.0.98.61:42243)
0924 07:14:02.001759 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:02.001761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:02.001909 (+ 148us) server_negotiation.cc:407] Connection header received
0924 07:14:02.001959 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:02.001960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:02.001970 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:02.002024 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:14:02.002163 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:02.002896 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:02.003725 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:02.003923 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:02.003992 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:02.004150 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:14:02.004182 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:14:02.004248 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:02.004289 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 07:14:02.004310 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 07:14:02.004920 23374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:02.002077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56594 (local address 127.0.98.60:37023)
0924 07:14:02.002120 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:02.002122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:02.002128 (+ 6us) server_negotiation.cc:407] Connection header received
0924 07:14:02.002150 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:02.002150 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:02.002159 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:02.002212 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:14:02.002327 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:02.003011 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:02.004049 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:02.004326 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:02.004558 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:02.004584 (+ 26us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:14:02.004611 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:14:02.004708 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:02.004831 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 07:14:02.004853 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 07:14:02.005172 23382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 07:14:02.001960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55074 (local address 127.0.98.62:40511)
0924 07:14:02.002097 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 07:14:02.002099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 07:14:02.002107 (+ 8us) server_negotiation.cc:407] Connection header received
0924 07:14:02.002151 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 07:14:02.002152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 07:14:02.002201 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 07:14:02.002276 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 07:14:02.002385 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:02.002860 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:02.003994 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 07:14:02.004334 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 07:14:02.004579 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 07:14:02.004648 (+ 69us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 07:14:02.004801 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 07:14:02.004948 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 07:14:02.004967 (+ 19us) server_negotiation.cc:299] Negotiation successful
0924 07:14:02.004995 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
I20260924 07:14:02.006793 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 22967
I20260924 07:14:02.012290 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 23042
I20260924 07:14:02.017120 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIclK1b/build/release/bin/kudu with pid 23117
2026-09-24T07:14:02Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2829 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5662 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (269616 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST